2024-03-29T10:23:30.726 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:23:30.726 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/orig.config.yaml 2024-03-29T10:23:30.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:23:30.851 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622638' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBx3xnHFbtJ25zGMuZ04L5GpSugqYgVK3jBBkv1e9NwE33Pz/SKbJjoF2qr6FsGun4tBX1imiEFrm8gjCWJMj7Q= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBY17LjcqhnrYdAm1fZk6fnGHYKtVuScVWQbsPTF73sszU4IAltJPgULm3CjqaD+OCMB9+GTv1bb7/2nO75Q508= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:23:30.851 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:23:30.866 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:23:30.866 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:23:30.868 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:23:30.893 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:23:30.893 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:23:30.893 INFO:teuthology.packaging:ref: None 2024-03-29T10:23:30.893 INFO:teuthology.packaging:tag: None 2024-03-29T10:23:30.893 INFO:teuthology.packaging:branch: main 2024-03-29T10:23:30.893 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:23:30.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:23:31.257 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:23:31.259 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:23:31.264 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:23:31.264 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:23:31.273 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:23:31.286 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:23:31.292 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:23:31.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:18.518909', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBx3xnHFbtJ25zGMuZ04L5GpSugqYgVK3jBBkv1e9NwE33Pz/SKbJjoF2qr6FsGun4tBX1imiEFrm8gjCWJMj7Q='} 2024-03-29T10:23:31.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:18.516968', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBY17LjcqhnrYdAm1fZk6fnGHYKtVuScVWQbsPTF73sszU4IAltJPgULm3CjqaD+OCMB9+GTv1bb7/2nO75Q508='} 2024-03-29T10:23:31.338 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:23:31.344 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:23:31.345 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:23:31.345 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:23:31.421 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd8a4a7f0d0>, signals=[15]) 2024-03-29T10:23:31.421 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:23:31.428 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:23:31.428 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:23:31.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:31.522 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:23:31.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:31.614 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:23:31.620 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-29T10:23:31.638 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-29T10:23:31.638 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-29T10:23:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:31.699 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:31.700 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-29T10:23:31.719 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-29T10:23:31.736 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-29T10:23:31.736 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-29T10:23:31.792 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-29T10:23:31.792 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:31.793 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:31.794 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-29T10:23:31.811 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:23:31.821 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:23:31.836 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:23:31.836 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:31.839 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:31.855 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:23:31.865 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:23:31.866 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:31.899 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:32.010 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:23:32.074 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:23:32.093 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:23:32.093 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:23:32.094 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:32.094 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:32.112 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-29T10:23:32.112 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-29T10:23:32.170 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:32.170 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-29T10:23:32.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-29T10:23:32.172 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:32.172 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-03-29T10:23:32.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-03-29T10:23:34.260 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:23:34.402 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:23:35.943 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-29T10:23:36.019 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:23:36.734 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-29T10:23:36.885 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T10:23:38.947 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T10:23:39.377 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T10:23:43.128 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T10:23:43.403 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T10:23:44.587 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-29T10:23:44.820 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-29T10:23:46.075 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T10:23:46.847 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:23:51.013 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:23:51.880 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T10:23:53.337 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:23:53.337 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:23:53.435 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:23:53.451 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:23:53.452 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:23:53.452 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-03-29T10:23:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:23:53.452 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 76 M 2024-03-29T10:23:53.452 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-03-29T10:23:53.453 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:23:53.615 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:23:54.145 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:23:54.145 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:23:54.223 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T10:23:54.237 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:23:54.249 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 76 M 2024-03-29T10:23:54.250 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-29T10:23:54.251 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:23:54.447 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 974 kB/s | 192 kB 00:00 2024-03-29T10:23:54.873 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T10:23:55.223 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T10:23:55.582 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T10:23:55.586 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:23:55.586 INFO:teuthology.orchestra.run.smithi151.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-29T10:23:55.586 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:23:55.983 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:23:55.983 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:23:56.023 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-03-29T10:23:56.281 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:23:56.285 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:23:56.285 INFO:teuthology.orchestra.run.smithi169.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-29T10:23:56.285 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:23:56.671 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:23:56.671 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:23:56.854 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:23:56.854 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:23:57.521 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:23:57.521 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:23:59.165 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:23:59.758 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:23:59.836 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:23:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:00.434 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:00.515 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:02.632 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:02.768 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:03.294 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:03.428 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:07.475 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:07.567 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:08.152 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:08.185 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:12.326 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:12.931 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:41.841 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:41.841 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:24:41.841 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:24:41.883 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.308 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.308 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:42.308 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:42.308 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:42.387 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.387 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:24:42.387 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:24:42.430 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.752 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.752 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:24:42.752 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:24:42.753 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:24:42.861 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:42.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:42.862 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:42.862 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:43.100 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:43.212 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.213 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.213 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.213 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:24:43.213 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:24:43.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:24:43.213 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:24:43.213 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:24:43.282 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:24:43.293 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.293 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:24:43.336 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:24:43.337 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:24:43.338 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:24:43.338 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:24:43.617 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:43.721 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.721 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.721 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.721 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:24:43.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:24:43.722 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:24:43.722 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:24:43.722 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:43.802 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:43.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-29T10:24:43.837 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:24:43.838 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-29T10:24:43.839 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:24:43.839 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:24:43.839 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:24:43.839 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:24:43.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:24:43.945 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-29T10:24:44.480 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-29T10:24:45.339 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:24:45.380 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-29T10:24:45.383 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-29T10:24:45.398 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.fCDXbsfTxZ 2024-03-29T10:24:45.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fCDXbsfTxZ 2024-03-29T10:24:45.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.fCDXbsfTxZ 2024-03-29T10:24:45.612 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.fCDXbsfTxZ is 6KB 2024-03-29T10:24:45.660 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.fCDXbsfTxZ 2024-03-29T10:24:45.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:24:45.739 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:24:45.739 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:24:45.739 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:24:45.739 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:24:45.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:24:45.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-03-29T10:24:45.930 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:24:45.955 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-29T10:24:45.957 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-29T10:24:45.973 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.x3aZevE3F0 2024-03-29T10:24:45.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x3aZevE3F0 2024-03-29T10:24:46.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.x3aZevE3F0 2024-03-29T10:24:46.179 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.x3aZevE3F0 is 6KB 2024-03-29T10:24:46.228 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.x3aZevE3F0 2024-03-29T10:24:46.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:24:46.308 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:24:46.308 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:24:46.308 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:24:46.308 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:24:46.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:24:46.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-29T10:25:15.930 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:25:15.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:25:15.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:16.445 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:25:16.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:25:16.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:34.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-29T10:25:35.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-29T10:25:43.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:25:43.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:44.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:25:44.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:46.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-29T10:25:47.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-29T10:25:58.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:25:58.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:59.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:25:59.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:16.991 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-29T10:26:32.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:26:32.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:32.375 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-29T10:26:32.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:26:32.765 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:26:32.765 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-29T10:26:32.779 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:26:32.780 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:26:32.780 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:26:32.780 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:26:33.781 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:26:33.781 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:26:33.849 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-29T10:26:33.897 DEBUG:teuthology.parallel:result is None 2024-03-29T10:26:59.074 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:27:14.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:27:14.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:27:14.432 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-29T10:27:14.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:27:14.796 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:27:14.796 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-29T10:27:14.811 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:27:14.812 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:27:14.812 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:27:14.812 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:27:15.813 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:27:15.814 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:27:15.875 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-29T10:27:15.909 DEBUG:teuthology.parallel:result is None 2024-03-29T10:27:15.910 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:27:15.937 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:27:15.938 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:27:15.941 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:27:15.958 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:27:15.985 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:27:16.010 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:27:16.010 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:27:16.014 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:27:16.045 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:27:16.077 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:27:16.077 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:27:16.080 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:27:16.120 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:16.124 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:16.134 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:16.138 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:16.140 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:27:16.175 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:27:16.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:27:16.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:27:16.207 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:27:16.229 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:27:16.230 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:27:16.249 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:27:16.265 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:16.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:16.398 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:16.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:16.532 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:27:16.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:27:16.603 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:16.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:16.712 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:16.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:16.849 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:27:16.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:27:16.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-29T10:27:16.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-29T10:27:16.983 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:27:17.007 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:27:17.452 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:27:17.460 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:27:17.460 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:27:17.473 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:27:17.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-29T10:27:17.572 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-29T10:27:17.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-29T10:27:17.635 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-29T10:27:17.637 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:27:17.637 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-29T10:27:17.659 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-29T10:27:17.659 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-29T10:27:17.682 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-29T10:27:17.682 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:27:17.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:27:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:17.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:27:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:17.758 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:27:17.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:27:17.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:27:17.843 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:27:17.853 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:27:17.873 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:27:17.874 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:31:40.681 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-03-29T10:31:40.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:31:40.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:40.780 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-29T10:31:40.857 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-29T10:31:40.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:31:40.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:40.948 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-29T10:31:40.989 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-29T10:31:40.990 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:31:41.001 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:31:41.001 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:41.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:41.004 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:41.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:41.032 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:41.049 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:41.061 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:41.077 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:41.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-29T10:31:41.091 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-29T10:31:41.106 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-29T10:31:41.108 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:41.120 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-29T10:31:41.124 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:41.137 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:41.153 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:41.179 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:41.182 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:41.182 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-29T10:31:41.182 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.182 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.182 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.183 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.205 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:41.207 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:41.208 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-29T10:31:41.208 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.208 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.208 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.208 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:41.209 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:31:41.217 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:31:41.217 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:31:41.217 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:31:41.218 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:31:41.223 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:31:41.223 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:31:41.224 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:41.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:41.226 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:41.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:41.377 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:41.377 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:41.442 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:41.443 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:41.680 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:41.681 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:31:41.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:41.695 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:41.695 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:31:41.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:41.709 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:41.709 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:41.724 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:41.724 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:41.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:41.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:41.859 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:41.874 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:41.924 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-29T10:31:41.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-29T10:31:41.940 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-03-29T10:31:41.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-03-29T10:31:42.175 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-29T10:31:42.190 INFO:teuthology.orchestra.run.smithi151.stdout:74 files removed 2024-03-29T10:31:42.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:42.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:43.388 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 95 kB/s | 82 kB 00:00 2024-03-29T10:31:43.588 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-29T10:31:44.139 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-29T10:31:44.881 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-29T10:31:45.467 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:31:46.664 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 3.3 kB/s | 10 kB 00:03 2024-03-29T10:31:47.200 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-29T10:31:47.306 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-29T10:31:47.967 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:31:47.991 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-29T10:31:49.408 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-29T10:31:50.391 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T10:31:52.042 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-29T10:31:52.058 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:02 2024-03-29T10:31:57.110 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-29T10:31:58.359 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-29T10:31:58.652 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-29T10:31:59.788 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-03-29T10:32:00.018 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T10:32:01.986 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:32:04.762 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T10:32:07.045 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-29T10:32:07.364 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:32:07.408 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:32:07.409 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:32:07.410 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-03-29T10:32:07.411 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:07.412 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2024-03-29T10:32:07.412 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:32:09.289 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.1 MB/s | 5.3 MB 00:01 2024-03-29T10:32:09.347 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T10:32:09.466 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T10:32:09.572 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.9 MB/s | 9.7 MB 00:01 2024-03-29T10:32:09.598 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.6 MB/s | 498 kB 00:00 2024-03-29T10:32:09.657 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 769 kB/s | 45 kB 00:00 2024-03-29T10:32:09.678 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:32:09.695 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:32:09.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse 2024-03-29T10:32:09.697 INFO:teuthology.orchestra.run.smithi151.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:32:09.698 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 58 M 2024-03-29T10:32:09.699 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:32:09.782 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.0 MB/s | 131 kB 00:00 2024-03-29T10:32:09.841 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T10:32:09.899 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T10:32:09.974 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T10:32:10.025 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:32:10.133 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2024-03-29T10:32:10.184 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T10:32:10.209 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T10:32:10.285 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 5.1 MB 00:00 2024-03-29T10:32:10.318 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-29T10:32:10.352 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-29T10:32:10.368 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 832 kB/s | 41 kB 00:00 2024-03-29T10:32:10.394 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:32:10.419 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T10:32:10.436 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T10:32:10.611 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T10:32:10.645 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-03-29T10:32:10.687 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T10:32:10.729 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:32:10.804 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-29T10:32:10.863 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-29T10:32:10.897 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T10:32:10.972 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T10:32:11.037 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 8.5 MB/s | 9.7 MB 00:01 2024-03-29T10:32:11.105 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.4 MB/s | 5.3 MB 00:01 2024-03-29T10:32:11.120 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 25 kB 00:00 2024-03-29T10:32:11.287 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.0 MB/s | 742 kB 00:00 2024-03-29T10:32:11.498 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.6 MB/s | 3.4 MB 00:00 2024-03-29T10:32:11.546 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.1 MB/s | 498 kB 00:00 2024-03-29T10:32:11.621 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 601 kB/s | 45 kB 00:00 2024-03-29T10:32:11.705 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T10:32:11.805 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T10:32:11.856 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 3.1 MB 00:00 2024-03-29T10:32:11.897 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 320 kB 00:00 2024-03-29T10:32:11.964 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 7.5 MB/s | 5.1 MB 00:00 2024-03-29T10:32:11.989 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.2 MB/s | 298 kB 00:00 2024-03-29T10:32:12.048 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:32:12.106 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 905 kB/s | 106 kB 00:00 2024-03-29T10:32:12.132 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T10:32:12.157 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T10:32:12.374 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.0 MB/s | 20 MB 00:02 2024-03-29T10:32:12.416 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 661 kB 00:00 2024-03-29T10:32:12.441 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 996 kB/s | 282 kB 00:00 2024-03-29T10:32:12.466 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 452 kB/s | 41 kB 00:00 2024-03-29T10:32:12.491 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-29T10:32:12.517 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:32:12.542 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 615 kB/s | 46 kB 00:00 2024-03-29T10:32:17.997 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 20 MB 00:10 2024-03-29T10:32:18.001 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:18.001 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.5 MB/s | 58 MB 00:10 2024-03-29T10:32:18.002 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:32:18.041 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:32:18.041 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:32:18.509 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:32:18.509 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:32:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:32:20.046 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:32:20.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:32:20.110 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:32:20.159 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:20.176 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:20.343 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:20.397 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:20.724 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:32:20.788 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:20.880 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:20.971 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:32:20.998 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:21.061 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:21.089 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:32:21.134 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:32:21.203 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:21.225 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:21.272 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:21.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:32:21.427 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:32:21.458 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:32:21.499 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:32:21.532 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:32:21.561 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:32:21.593 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:32:21.918 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:32:21.948 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:22.038 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:22.070 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:32:22.118 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:32:22.569 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:32:22.601 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:22.672 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:22.741 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:32:22.770 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:32:24.901 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:24.942 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:25.514 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:25.779 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:25.864 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:32:25.864 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:32:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:27.940 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 20 kB/s | 309 kB 00:15 2024-03-29T10:32:27.991 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-03-29T10:32:28.049 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T10:32:28.158 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-29T10:32:28.225 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T10:32:28.367 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T10:32:28.984 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.5 MB/s | 3.4 MB 00:00 2024-03-29T10:32:29.259 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-29T10:32:35.320 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-29T10:32:35.321 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-29T10:32:35.321 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:36.215 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:36.239 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:36.239 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:36.239 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:32:36.240 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:36.240 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:36.240 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:36.241 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:36.309 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:36.310 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:37.473 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:37.474 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:32:37.475 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:32:37.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:32:37.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:32:37.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:32:37.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:37.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:32:37.477 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:32:38.398 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:32:38.399 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.400 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:38.401 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:32:38.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-03-29T10:32:39.312 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:35 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:32:39.470 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:32:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-29T10:32:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:39.473 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-29T10:32:39.473 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 204 M 2024-03-29T10:32:39.473 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:32:39.747 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T10:32:39.798 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:32:39.823 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T10:32:39.848 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 635 kB/s | 163 kB 00:00 2024-03-29T10:32:40.682 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 218 kB/s | 187 kB 00:00 2024-03-29T10:32:41.932 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T10:32:41.935 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:41.935 INFO:teuthology.orchestra.run.smithi169.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T10:32:41.951 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:32:41.972 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:32:41.973 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:32:42.352 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:32:42.353 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:32:42.927 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:32:42.990 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:32:43.050 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:32:43.084 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:43.149 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:32:43.177 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:47.730 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:47.810 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:32:50.375 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:50.788 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:32:50.789 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:32:50.789 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:32:50.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:50.789 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:32:51.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-03-29T10:32:51.670 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:47 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:32:51.829 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:32:51.831 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-29T10:32:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:32:51.831 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:32:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:32:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:32:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:32:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:51.835 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:32:51.835 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-29T10:32:51.835 INFO:teuthology.orchestra.run.smithi169.stdout:Install 36 Packages 2024-03-29T10:32:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:32:51.836 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 30 M 2024-03-29T10:32:51.836 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 107 M 2024-03-29T10:32:51.836 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:32:52.802 INFO:teuthology.orchestra.run.smithi169.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-29T10:32:53.211 INFO:teuthology.orchestra.run.smithi169.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-29T10:32:53.286 INFO:teuthology.orchestra.run.smithi169.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-29T10:32:53.369 INFO:teuthology.orchestra.run.smithi169.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-03-29T10:32:53.395 INFO:teuthology.orchestra.run.smithi169.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-29T10:32:53.520 INFO:teuthology.orchestra.run.smithi169.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:32:53.546 INFO:teuthology.orchestra.run.smithi169.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T10:32:53.571 INFO:teuthology.orchestra.run.smithi169.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-29T10:32:53.596 INFO:teuthology.orchestra.run.smithi169.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 946 kB/s | 47 kB 00:00 2024-03-29T10:32:53.621 INFO:teuthology.orchestra.run.smithi169.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-29T10:32:53.647 INFO:teuthology.orchestra.run.smithi169.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:32:53.688 INFO:teuthology.orchestra.run.smithi169.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:32:53.714 INFO:teuthology.orchestra.run.smithi169.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-29T10:32:53.739 INFO:teuthology.orchestra.run.smithi169.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:32:53.764 INFO:teuthology.orchestra.run.smithi169.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:32:53.814 INFO:teuthology.orchestra.run.smithi169.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-29T10:32:54.115 INFO:teuthology.orchestra.run.smithi169.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T10:32:54.140 INFO:teuthology.orchestra.run.smithi169.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 312 kB/s | 125 kB 00:00 2024-03-29T10:32:54.199 INFO:teuthology.orchestra.run.smithi169.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 393 kB/s | 151 kB 00:00 2024-03-29T10:32:54.274 INFO:teuthology.orchestra.run.smithi169.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-03-29T10:32:54.349 INFO:teuthology.orchestra.run.smithi169.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T10:32:54.383 INFO:teuthology.orchestra.run.smithi169.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 708 kB/s | 172 kB 00:00 2024-03-29T10:32:54.408 INFO:teuthology.orchestra.run.smithi169.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 300 kB/s | 18 kB 00:00 2024-03-29T10:32:54.442 INFO:teuthology.orchestra.run.smithi169.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-29T10:32:54.467 INFO:teuthology.orchestra.run.smithi169.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-03-29T10:32:54.501 INFO:teuthology.orchestra.run.smithi169.stdout:(26/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-29T10:32:54.526 INFO:teuthology.orchestra.run.smithi169.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-29T10:32:54.551 INFO:teuthology.orchestra.run.smithi169.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-03-29T10:32:54.593 INFO:teuthology.orchestra.run.smithi169.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 856 kB/s | 79 kB 00:00 2024-03-29T10:32:54.627 INFO:teuthology.orchestra.run.smithi169.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-29T10:32:54.669 INFO:teuthology.orchestra.run.smithi169.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-29T10:32:54.694 INFO:teuthology.orchestra.run.smithi169.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 894 kB/s | 90 kB 00:00 2024-03-29T10:32:54.714 INFO:teuthology.orchestra.run.smithi169.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-29T10:32:54.745 INFO:teuthology.orchestra.run.smithi169.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:32:54.787 INFO:teuthology.orchestra.run.smithi169.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-29T10:32:54.853 INFO:teuthology.orchestra.run.smithi169.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-29T10:32:54.857 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:54.857 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-29T10:32:54.883 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:32:54.926 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:32:54.926 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:32:55.335 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:32:55.336 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:32:56.704 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:32:56.764 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:32:56.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:32:56.850 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:32:56.920 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:32:56.999 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:32:57.039 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:32:57.146 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:32:57.235 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:32:57.284 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:32:57.364 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:32:57.418 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:32:57.456 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:32:57.506 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:32:57.561 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:32:57.746 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:32:57.803 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:32:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:32:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:32:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:32:57.973 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:32:58.017 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:32:58.058 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:32:58.098 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:32:58.134 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:32:58.168 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:32:58.258 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:32:58.299 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:32:58.370 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:32:58.514 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:32:58.642 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:32:58.801 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:32:58.829 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:58.855 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:00.305 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:00.332 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:00.332 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:00.332 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:33:00.332 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:00.333 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:00.333 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:00.773 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:00.796 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:00.796 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:00.796 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:33:00.796 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:00.796 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:00.797 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:00.998 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:01.022 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:01.022 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:01.022 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:33:01.022 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:01.023 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:01.023 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:33:02.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:33:02.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:33:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:33:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:33:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:33:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:33:02.951 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:33:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:33:02.953 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:33:02.953 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:33:02.953 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:33:03.897 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:33:03.897 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:33:03.898 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:33:03.899 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:03.900 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:04.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-03-29T10:33:04.833 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:04.932 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:04.984 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:04.985 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:04.985 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:05.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-03-29T10:33:05.635 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:05.786 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:05.787 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:05.787 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:05.787 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 747 k 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 754 k 2024-03-29T10:33:05.788 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:06.241 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T10:33:06.241 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:06.241 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T10:33:06.242 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:06.247 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:06.247 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:06.252 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:06.253 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:06.327 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:06.739 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:06.820 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:07.288 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:07.595 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:07.595 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:07.595 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:07.596 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:07.596 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:07.596 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:07.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:33:08.347 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:08.498 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:08.498 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:33:08.499 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:08.500 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-03-29T10:33:08.500 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 436 k 2024-03-29T10:33:08.500 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:08.813 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 464 kB/s | 145 kB 00:00 2024-03-29T10:33:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:Total 462 kB/s | 145 kB 00:00 2024-03-29T10:33:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:08.822 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:08.822 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:08.874 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:08.874 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:09.016 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:09.098 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:09.122 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:09.122 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:09.122 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:33:09.123 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:10.045 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:10.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-03-29T10:33:10.897 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:10.997 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:11.048 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:11.050 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:11.050 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:11.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:33:11.669 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:11.819 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:11.820 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:33:11.821 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-29T10:33:11.822 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:11.823 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.9 M 2024-03-29T10:33:11.823 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-29T10:33:11.823 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:12.659 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-29T10:33:12.718 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 124 kB/s | 31 kB 00:00 2024-03-29T10:33:12.793 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.1 MB/s | 238 kB 00:00 2024-03-29T10:33:12.835 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-29T10:33:12.885 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.2 MB/s | 1.8 MB 00:00 2024-03-29T10:33:13.002 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-29T10:33:13.736 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:33:13.803 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-29T10:33:13.862 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-29T10:33:13.970 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T10:33:14.503 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 505 kB/s | 863 kB 00:01 2024-03-29T10:33:16.295 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 158 kB/s | 548 kB 00:03 2024-03-29T10:33:16.299 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:16.299 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.5 MB/s | 6.9 MB 00:04 2024-03-29T10:33:16.319 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:16.339 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:16.339 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:16.521 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:16.522 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:17.045 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:17.145 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:33:17.189 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:33:17.244 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:33:17.283 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:33:17.387 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:33:17.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:33:17.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:33:17.654 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:33:17.683 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:17.715 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:33:17.956 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:33:17.990 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:33:18.861 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:18.862 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:33:18.862 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:33:19.211 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:19.212 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:19.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:33:20.028 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:20.181 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:33:20.182 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-29T10:33:20.183 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:20.184 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-03-29T10:33:20.184 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-03-29T10:33:20.184 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:20.477 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2024-03-29T10:33:20.510 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-29T10:33:20.552 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 34 kB 00:00 2024-03-29T10:33:20.611 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:33:20.778 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T10:33:20.845 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:33:20.870 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:33:21.120 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 9.4 MB/s | 7.4 MB 00:00 2024-03-29T10:33:21.288 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-29T10:33:21.479 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T10:33:22.155 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 189 kB/s | 195 kB 00:01 2024-03-29T10:33:22.663 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 448 kB/s | 803 kB 00:01 2024-03-29T10:33:22.666 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:22.666 INFO:teuthology.orchestra.run.smithi169.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-29T10:33:22.710 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:22.736 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:22.736 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:23.098 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:23.098 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:23.798 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:23.885 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:33:23.933 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:33:24.034 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:24.070 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:33:24.247 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:24.278 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:33:24.990 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:33:25.122 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:33:26.371 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:33:27.395 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:33:27.428 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:33:30.149 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:33:30.150 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:33:30.150 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:33:30.584 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:30.585 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:33:30.586 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:30.586 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:30.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:33:31.535 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:31.687 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:31.688 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:31.688 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:31.688 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:31.690 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2024-03-29T10:33:31.691 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-29T10:33:31.691 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:31.862 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2024-03-29T10:33:31.887 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T10:33:31.937 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:33:31.963 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:33:31.989 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:33:32.013 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-29T10:33:32.038 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 175 kB/s | 52 kB 00:00 2024-03-29T10:33:32.230 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-29T10:33:32.255 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T10:33:32.364 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 623 kB/s | 203 kB 00:00 2024-03-29T10:33:32.423 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T10:33:32.481 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-03-29T10:33:32.515 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-29T10:33:32.519 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:32.519 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-29T10:33:32.519 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:32.533 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:32.533 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:32.623 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:32.623 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:33.125 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:33.313 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:33:33.366 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:33:33.438 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:33:33.475 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:33:33.517 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:33:33.619 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:33:33.719 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:33:33.777 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:33:34.502 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:33:34.557 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:33:34.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:33:34.648 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:33:34.678 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:33:35.486 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:33:35.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:33:35.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:33:35.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:33:35.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:33:35.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:33:35.852 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:35.853 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:36.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:33:36.693 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:36.847 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:33:36.848 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-29T10:33:36.849 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:36.850 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2024-03-29T10:33:36.850 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-03-29T10:33:36.850 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:37.636 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 760 kB/s | 248 kB 00:00 2024-03-29T10:33:37.720 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 336 kB/s | 138 kB 00:00 2024-03-29T10:33:37.820 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-29T10:33:38.003 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 204 kB/s | 58 kB 00:00 2024-03-29T10:33:38.037 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-03-29T10:33:38.087 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:33:38.089 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:38.089 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-29T10:33:38.102 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:38.113 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:38.113 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:38.204 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:38.205 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:38.679 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:33:38.730 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:33:39.259 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:33:39.341 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:33:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:33:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:40.245 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:40.245 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:33:40.245 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:33:40.246 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:33:40.246 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:33:40.246 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:33:40.551 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:33:40.552 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:33:40.552 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:40.552 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:40.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-03-29T10:33:41.341 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:41.491 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:41.491 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:41.492 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:33:41.493 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:41.493 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:41.493 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:41.493 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-29T10:33:41.493 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:41.494 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 964 k 2024-03-29T10:33:41.494 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.8 M 2024-03-29T10:33:41.494 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:41.664 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 461 kB/s | 79 kB 00:00 2024-03-29T10:33:42.664 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 756 kB/s | 885 kB 00:01 2024-03-29T10:33:42.665 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:42.666 INFO:teuthology.orchestra.run.smithi169.stdout:Total 821 kB/s | 964 kB 00:01 2024-03-29T10:33:42.666 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:42.687 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:42.687 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:42.762 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:42.762 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:42.954 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:43.106 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:33:43.172 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:43.883 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:43.883 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:44.199 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:44.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-03-29T10:33:45.085 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:45.242 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 332 k 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.5 M 2024-03-29T10:33:45.244 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:45.369 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 553 kB/s | 69 kB 00:00 2024-03-29T10:33:45.686 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 598 kB/s | 264 kB 00:00 2024-03-29T10:33:45.690 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:45.690 INFO:teuthology.orchestra.run.smithi169.stdout:Total 749 kB/s | 332 kB 00:00 2024-03-29T10:33:45.690 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:45.696 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:45.696 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:45.733 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:45.734 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:45.882 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:46.061 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:33:46.113 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:46.129 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:46.130 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:46.130 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:33:46.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:46.881 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:33:47.246 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:47.247 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:47.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-03-29T10:33:48.174 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:48.324 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:48.325 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:48.325 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-29T10:33:48.325 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:48.325 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:48.325 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 127 k 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-03-29T10:33:48.326 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:48.733 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 311 kB/s | 127 kB 00:00 2024-03-29T10:33:48.734 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:48.734 INFO:teuthology.orchestra.run.smithi169.stdout:Total 310 kB/s | 127 kB 00:00 2024-03-29T10:33:48.734 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:48.740 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:48.740 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:48.797 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:48.797 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:48.942 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:49.056 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:49.521 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:49.829 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:49.830 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:49.830 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:49.830 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:49.830 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:49.830 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:50.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-03-29T10:33:50.627 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:50.728 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:50.783 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:50.784 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:50.784 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:50.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-03-29T10:33:51.430 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:51.580 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:33:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:51.582 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 k 2024-03-29T10:33:51.582 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 140 k 2024-03-29T10:33:51.582 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:51.836 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 123 kB/s | 31 kB 00:00 2024-03-29T10:33:51.837 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:51.837 INFO:teuthology.orchestra.run.smithi169.stdout:Total 122 kB/s | 31 kB 00:00 2024-03-29T10:33:51.837 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:51.841 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:51.841 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:51.866 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:51.867 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:52.069 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:52.745 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:52.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-03-29T10:33:53.467 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:53.565 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:53.616 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:53.617 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:53.617 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:53.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-03-29T10:33:54.233 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:54.332 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:54.384 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:54.385 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:54.385 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:54.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-03-29T10:33:55.005 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:55.106 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:55.159 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:55.160 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:55.160 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:55.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-03-29T10:33:55.779 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:55.879 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:55.930 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:55.931 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:55.931 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:55.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-03-29T10:33:56.555 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:56.657 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:56.709 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:56.710 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:56.710 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:56.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-03-29T10:33:57.340 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:57.448 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:57.504 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:57.505 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-29T10:33:57.505 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:57.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-03-29T10:33:58.152 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:33:58.308 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:33:58.309 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:58.310 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-03-29T10:33:58.310 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-03-29T10:33:58.310 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:33:58.523 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 401 kB/s | 85 kB 00:00 2024-03-29T10:33:58.523 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:58.524 INFO:teuthology.orchestra.run.smithi169.stdout:Total 397 kB/s | 85 kB 00:00 2024-03-29T10:33:58.524 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:33:58.531 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:33:58.531 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:33:58.588 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:33:58.588 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:33:58.725 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:33:58.831 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:59.224 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:33:59.506 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:33:59.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-03-29T10:34:00.344 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:34:00.500 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:34:00.500 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.1 M 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 13 M 2024-03-29T10:34:00.501 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:34:01.018 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.0 MB/s | 3.1 MB 00:00 2024-03-29T10:34:01.018 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:01.019 INFO:teuthology.orchestra.run.smithi169.stdout:Total 6.0 MB/s | 3.1 MB 00:00 2024-03-29T10:34:01.019 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:34:01.028 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:34:01.028 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:34:01.102 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:34:01.103 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:34:01.533 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:34:01.594 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:01.619 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:01.619 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:01.619 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:34:01.619 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:01.620 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:01.620 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:02.796 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:34:03.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-03-29T10:34:03.613 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 10:32:04 AM UTC. 2024-03-29T10:34:03.761 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-29T10:34:03.762 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:03.762 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-29T10:34:03.762 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 171 k 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 494 k 2024-03-29T10:34:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-29T10:34:03.971 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 826 kB/s | 171 kB 00:00 2024-03-29T10:34:03.972 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:03.972 INFO:teuthology.orchestra.run.smithi169.stdout:Total 820 kB/s | 171 kB 00:00 2024-03-29T10:34:03.972 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-29T10:34:03.979 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-29T10:34:03.979 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-29T10:34:04.035 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-29T10:34:04.035 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-29T10:34:04.180 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-29T10:34:04.278 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:04.710 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:34:05.040 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-29T10:34:05.282 DEBUG:teuthology.parallel:result is None 2024-03-29T10:34:21.839 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] libarrow-9.0.0-10.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/l/libarrow-9.0.0-10.el9.x86_64.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T10:34:21.840 INFO:teuthology.orchestra.run.smithi151.stdout:[MIRROR] libarrow-doc-9.0.0-10.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-10.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T10:34:22.651 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 35 kB/s | 4.4 MB 02:10 2024-03-29T10:34:25.426 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 200 B/s | 26 kB 02:12 2024-03-29T10:34:25.428 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:25.428 INFO:teuthology.orchestra.run.smithi151.stdout:Total 436 kB/s | 58 MB 02:15 2024-03-29T10:34:25.429 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:34:25.468 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:34:25.468 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:34:25.942 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:34:25.942 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:34:27.426 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:34:27.501 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:34:27.530 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:34:27.555 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:34:27.587 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:34:27.603 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:34:27.770 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:34:27.825 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:34:28.142 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:34:28.224 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:34:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:34:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:34:28.516 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:34:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:34:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:34:28.644 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:34:28.691 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:34:28.726 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:34:28.833 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:34:28.876 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:34:28.917 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:34:28.950 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:34:28.979 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:34:29.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:34:29.323 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:34:29.350 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:34:29.438 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:34:29.499 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:34:29.549 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:34:30.002 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:34:30.044 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:34:30.097 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:34:30.166 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:34:30.196 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:34:32.365 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:32.957 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:32.995 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:34:33.082 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:34:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:34:33.082 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:34:33.082 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:42.377 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-03-29T10:34:42.377 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-03-29T10:34:42.377 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-03-29T10:34:42.378 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-03-29T10:34:42.378 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-03-29T10:34:42.378 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-03-29T10:34:42.378 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-03-29T10:34:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:43.271 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:34:43.296 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:43.298 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:34:43.372 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:34:43.372 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:44.526 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:44.526 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:34:44.526 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:34:44.526 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:34:44.526 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:34:44.527 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:34:44.528 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:34:44.529 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:34:44.529 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:34:44.529 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:34:44.529 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:44.530 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:34:44.530 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:34:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:34:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:45.570 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:34:45.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-03-29T10:34:46.439 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:34:46.598 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:34:46.598 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:34:46.598 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-29T10:34:46.599 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:46.600 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 48 M 2024-03-29T10:34:46.600 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 204 M 2024-03-29T10:34:46.600 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:34:47.031 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:34:47.081 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:34:47.115 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:34:47.240 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:34:47.266 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 414 kB/s | 163 kB 00:00 2024-03-29T10:34:49.216 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T10:34:49.219 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:49.219 INFO:teuthology.orchestra.run.smithi151.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T10:34:49.234 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:34:49.256 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:34:49.256 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:34:49.628 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:34:49.629 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:34:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:34:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:34:50.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:34:50.358 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:34:50.400 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:34:50.428 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:34:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:34:55.096 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:34:57.685 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:34:57.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:34:57.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:34:57.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:34:57.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:34:57.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:34:58.047 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:34:58.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:58.047 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:58.048 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:34:58.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-03-29T10:34:58.957 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:02:51 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:34:59.128 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:34:59.131 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:34:59.132 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:34:59.133 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout:========================================================================================= 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout:Install 36 Packages 2024-03-29T10:34:59.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:34:59.135 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-03-29T10:34:59.135 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 107 M 2024-03-29T10:34:59.135 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:34:59.534 INFO:teuthology.orchestra.run.smithi151.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T10:34:59.892 INFO:teuthology.orchestra.run.smithi151.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-03-29T10:35:00.484 INFO:teuthology.orchestra.run.smithi151.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-29T10:35:00.726 INFO:teuthology.orchestra.run.smithi151.stdout:(4/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-29T10:35:00.751 INFO:teuthology.orchestra.run.smithi151.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 926 kB/s | 247 kB 00:00 2024-03-29T10:35:00.818 INFO:teuthology.orchestra.run.smithi151.stdout:(6/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-29T10:35:00.961 INFO:teuthology.orchestra.run.smithi151.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 797 kB/s | 187 kB 00:00 2024-03-29T10:35:00.986 INFO:teuthology.orchestra.run.smithi151.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 222 kB/s | 32 kB 00:00 2024-03-29T10:35:01.011 INFO:teuthology.orchestra.run.smithi151.stdout:(9/36): python3-mako-1.1.4-6.el9.noarch.rpm 610 kB/s | 144 kB 00:00 2024-03-29T10:35:01.036 INFO:teuthology.orchestra.run.smithi151.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 627 kB/s | 47 kB 00:00 2024-03-29T10:35:01.062 INFO:teuthology.orchestra.run.smithi151.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 510 kB/s | 39 kB 00:00 2024-03-29T10:35:01.087 INFO:teuthology.orchestra.run.smithi151.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-29T10:35:01.196 INFO:teuthology.orchestra.run.smithi151.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 7.2 MB/s | 1.1 MB 00:00 2024-03-29T10:35:01.222 INFO:teuthology.orchestra.run.smithi151.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 648 kB/s | 103 kB 00:00 2024-03-29T10:35:01.246 INFO:teuthology.orchestra.run.smithi151.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 780 kB/s | 124 kB 00:00 2024-03-29T10:35:01.272 INFO:teuthology.orchestra.run.smithi151.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-29T10:35:01.464 INFO:teuthology.orchestra.run.smithi151.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 789 kB/s | 151 kB 00:00 2024-03-29T10:35:01.489 INFO:teuthology.orchestra.run.smithi151.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 516 kB/s | 125 kB 00:00 2024-03-29T10:35:01.523 INFO:teuthology.orchestra.run.smithi151.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 751 kB/s | 43 kB 00:00 2024-03-29T10:35:01.548 INFO:teuthology.orchestra.run.smithi151.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-03-29T10:35:01.615 INFO:teuthology.orchestra.run.smithi151.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T10:35:01.633 INFO:teuthology.orchestra.run.smithi151.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 127 kB/s | 11 kB 00:00 2024-03-29T10:35:01.658 INFO:teuthology.orchestra.run.smithi151.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-29T10:35:01.675 INFO:teuthology.orchestra.run.smithi151.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 298 kB/s | 18 kB 00:00 2024-03-29T10:35:01.692 INFO:teuthology.orchestra.run.smithi151.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 392 kB/s | 23 kB 00:00 2024-03-29T10:35:01.709 INFO:teuthology.orchestra.run.smithi151.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 384 kB/s | 19 kB 00:00 2024-03-29T10:35:01.726 INFO:teuthology.orchestra.run.smithi151.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-29T10:35:01.751 INFO:teuthology.orchestra.run.smithi151.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 782 kB/s | 46 kB 00:00 2024-03-29T10:35:01.776 INFO:teuthology.orchestra.run.smithi151.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T10:35:01.802 INFO:teuthology.orchestra.run.smithi151.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T10:35:01.819 INFO:teuthology.orchestra.run.smithi151.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 243 kB/s | 16 kB 00:00 2024-03-29T10:35:01.844 INFO:teuthology.orchestra.run.smithi151.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T10:35:01.861 INFO:teuthology.orchestra.run.smithi151.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 607 kB/s | 36 kB 00:00 2024-03-29T10:35:01.912 INFO:teuthology.orchestra.run.smithi151.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T10:35:01.929 INFO:teuthology.orchestra.run.smithi151.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 297 kB/s | 20 kB 00:00 2024-03-29T10:35:01.954 INFO:teuthology.orchestra.run.smithi151.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T10:35:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:35:01.979 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:02.019 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:02.019 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:02.433 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:02.433 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:03.779 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:03.840 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:35:03.873 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:35:03.901 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:35:03.971 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:35:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:35:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:35:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:35:04.211 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:35:04.253 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:35:04.333 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:35:04.387 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:35:04.426 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:35:04.474 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:35:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:35:04.721 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:35:04.765 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:35:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:35:04.830 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:35:04.860 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:35:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:35:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:35:05.005 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:35:05.037 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:35:05.072 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:35:05.107 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:35:05.195 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:35:05.238 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:35:05.309 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:35:05.445 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:35:05.564 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:35:05.722 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:35:05.761 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:35:05.786 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:35:05.787 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:05.787 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:35:05.787 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:35:05.787 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:35:05.787 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:07.221 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:35:07.247 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:35:07.248 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:07.248 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:35:07.248 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:35:07.248 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:35:07.248 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:07.689 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:35:07.712 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:07.914 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:35:07.938 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:08.003 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:35:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:35:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:35:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:35:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:35:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:35:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:35:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:35:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:35:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:35:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:11.025 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:11.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-03-29T10:35:11.952 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:04 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:12.050 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:12.102 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:12.103 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:35:12.103 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:12.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-03-29T10:35:12.741 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:12.891 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:12.892 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 747 k 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 754 k 2024-03-29T10:35:12.893 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:13.204 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-29T10:35:13.205 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:13.205 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.3 MB/s | 747 kB 00:00 2024-03-29T10:35:13.205 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:13.210 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:13.210 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:13.215 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:13.216 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:13.293 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:13.860 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:14.225 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:14.550 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:14.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:14.550 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:14.551 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:14.551 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:14.551 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:14.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:35:15.331 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:08 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:15.485 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:35:15.486 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 436 k 2024-03-29T10:35:15.487 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:15.706 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 663 kB/s | 145 kB 00:00 2024-03-29T10:35:15.707 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:15.707 INFO:teuthology.orchestra.run.smithi151.stdout:Total 659 kB/s | 145 kB 00:00 2024-03-29T10:35:15.708 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:15.715 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:15.715 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:15.767 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:15.767 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:15.908 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:15.974 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:15.999 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:15.999 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:15.999 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:35:15.999 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:17.016 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:17.017 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:17.017 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:17.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-03-29T10:35:17.841 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:10 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:17.939 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:17.992 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:17.993 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:35:17.993 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:18.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:35:18.626 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:11 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:18.777 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:18.778 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-29T10:35:18.778 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:18.778 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-29T10:35:18.778 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:18.779 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:====================================================================================== 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.9 M 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-29T10:35:18.780 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:19.547 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 43 kB/s | 15 kB 00:00 2024-03-29T10:35:19.689 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 62 kB/s | 31 kB 00:00 2024-03-29T10:35:19.798 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-29T10:35:19.832 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-29T10:35:20.007 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-29T10:35:20.124 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T10:35:20.149 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-29T10:35:20.291 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-29T10:35:20.317 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 865 kB/s | 144 kB 00:00 2024-03-29T10:35:20.350 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 523 kB/s | 31 kB 00:00 2024-03-29T10:35:20.384 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.5 MB/s | 1.8 MB 00:01 2024-03-29T10:35:20.418 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T10:35:20.421 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:20.421 INFO:teuthology.orchestra.run.smithi151.stdout:Total 4.2 MB/s | 6.9 MB 00:01 2024-03-29T10:35:20.440 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:20.457 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:20.457 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:20.621 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:20.621 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:21.127 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:35:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:35:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:35:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:35:21.435 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:35:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:35:21.652 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:35:21.704 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:35:21.744 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:35:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:35:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:35:22.059 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:35:22.929 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:35:22.929 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:35:22.930 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:35:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:35:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:23.292 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:23.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:35:24.106 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:17 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:24.259 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:35:24.260 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================================ 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout:Install 12 Packages 2024-03-29T10:35:24.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:24.262 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 37 M 2024-03-29T10:35:24.262 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-03-29T10:35:24.262 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:24.689 INFO:teuthology.orchestra.run.smithi151.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 160 kB/s | 32 kB 00:00 2024-03-29T10:35:24.748 INFO:teuthology.orchestra.run.smithi151.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 294 kB/s | 17 kB 00:00 2024-03-29T10:35:24.815 INFO:teuthology.orchestra.run.smithi151.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-03-29T10:35:24.982 INFO:teuthology.orchestra.run.smithi151.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-03-29T10:35:25.132 INFO:teuthology.orchestra.run.smithi151.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-14 12 MB/s | 7.4 MB 00:00 2024-03-29T10:35:25.241 INFO:teuthology.orchestra.run.smithi151.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-29T10:35:25.367 INFO:teuthology.orchestra.run.smithi151.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-29T10:35:25.400 INFO:teuthology.orchestra.run.smithi151.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 820 kB/s | 217 kB 00:00 2024-03-29T10:35:25.825 INFO:teuthology.orchestra.run.smithi151.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-03-29T10:35:25.951 INFO:teuthology.orchestra.run.smithi151.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 356 kB/s | 195 kB 00:00 2024-03-29T10:35:25.976 INFO:teuthology.orchestra.run.smithi151.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-29T10:35:26.010 INFO:teuthology.orchestra.run.smithi151.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-29T10:35:26.012 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:26.012 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T10:35:26.052 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:26.075 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:26.075 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:26.449 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:26.449 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:35:27.264 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:35:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:35:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:35:27.582 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:35:27.617 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:35:27.697 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:35:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:35:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:35:29.720 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:35:30.758 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:35:30.789 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:35:33.359 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:35:33.359 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:35:33.359 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:35:33.360 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:35:33.825 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:35:33.825 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:33.825 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:33.825 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:33.826 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:34.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:35:34.781 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:27 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:34.933 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:34.934 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:34.934 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:34.934 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:34.934 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:34.934 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:35:34.935 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:34.936 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:34.936 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:34.936 INFO:teuthology.orchestra.run.smithi151.stdout:Install 13 Packages 2024-03-29T10:35:34.936 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:34.937 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 2.0 M 2024-03-29T10:35:34.937 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 26 M 2024-03-29T10:35:34.937 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:35.204 INFO:teuthology.orchestra.run.smithi151.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-29T10:35:35.220 INFO:teuthology.orchestra.run.smithi151.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-29T10:35:35.279 INFO:teuthology.orchestra.run.smithi151.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:35:35.304 INFO:teuthology.orchestra.run.smithi151.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T10:35:35.330 INFO:teuthology.orchestra.run.smithi151.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 191 kB/s | 52 kB 00:00 2024-03-29T10:35:35.355 INFO:teuthology.orchestra.run.smithi151.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:35:35.380 INFO:teuthology.orchestra.run.smithi151.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 566 kB/s | 43 kB 00:00 2024-03-29T10:35:35.623 INFO:teuthology.orchestra.run.smithi151.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T10:35:35.656 INFO:teuthology.orchestra.run.smithi151.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-29T10:35:35.807 INFO:teuthology.orchestra.run.smithi151.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-29T10:35:35.832 INFO:teuthology.orchestra.run.smithi151.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 449 kB/s | 203 kB 00:00 2024-03-29T10:35:35.924 INFO:teuthology.orchestra.run.smithi151.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 766 kB/s | 90 kB 00:00 2024-03-29T10:35:35.957 INFO:teuthology.orchestra.run.smithi151.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T10:35:35.961 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:35.962 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-29T10:35:35.962 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:35.975 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:35.975 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:36.067 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:36.067 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:36.582 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:36.755 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:35:36.809 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:35:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:35:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:35:36.952 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:35:37.046 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:35:37.156 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:35:37.225 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:35:37.968 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:35:38.011 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:35:38.045 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:35:38.093 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:35:38.115 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:35:38.905 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:35:38.906 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:35:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:35:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:39.264 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:39.265 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:39.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:35:40.100 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:33 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:40.252 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:40.253 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:40.253 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:40.253 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:40.253 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout:======================================================================================== 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout:Install 6 Packages 2024-03-29T10:35:40.254 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:40.255 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 6.8 M 2024-03-29T10:35:40.255 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 33 M 2024-03-29T10:35:40.255 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:40.917 INFO:teuthology.orchestra.run.smithi151.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T10:35:40.931 INFO:teuthology.orchestra.run.smithi151.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 446 kB/s | 138 kB 00:00 2024-03-29T10:35:40.974 INFO:teuthology.orchestra.run.smithi151.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 707 kB/s | 248 kB 00:00 2024-03-29T10:35:41.141 INFO:teuthology.orchestra.run.smithi151.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 276 kB/s | 58 kB 00:00 2024-03-29T10:35:42.216 INFO:teuthology.orchestra.run.smithi151.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 421 kB/s | 548 kB 00:01 2024-03-29T10:35:44.232 INFO:teuthology.orchestra.run.smithi151.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 16 kB/s | 51 kB 00:03 2024-03-29T10:35:44.233 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:44.234 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.7 MB/s | 6.8 MB 00:03 2024-03-29T10:35:44.249 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:44.261 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:44.261 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:44.352 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:44.352 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:44.666 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:44.849 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:35:44.910 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:35:45.429 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:35:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:35:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:35:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:35:46.396 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:35:46.396 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:35:46.396 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:35:46.397 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:35:46.397 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:35:46.397 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:35:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:46.697 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:46.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-03-29T10:35:47.501 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:40 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:47.650 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:47.650 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:47.650 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-29T10:35:47.651 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:47.652 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 964 k 2024-03-29T10:35:47.652 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.8 M 2024-03-29T10:35:47.652 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:47.786 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 585 kB/s | 79 kB 00:00 2024-03-29T10:35:48.103 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T10:35:48.105 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:48.105 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T10:35:48.105 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:48.125 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:48.125 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:48.197 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:48.198 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:48.380 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:48.521 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:35:48.603 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:35:49.343 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:35:49.343 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:35:49.618 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:35:49.618 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:49.619 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:49.619 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:35:49.619 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:49.619 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:49.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-volume 2024-03-29T10:35:50.520 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:43 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-03-29T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 332 k 2024-03-29T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 1.5 M 2024-03-29T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:50.812 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 518 kB/s | 69 kB 00:00 2024-03-29T10:35:50.962 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 932 kB/s | 264 kB 00:00 2024-03-29T10:35:50.963 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:50.964 INFO:teuthology.orchestra.run.smithi151.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T10:35:50.964 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:50.972 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:50.972 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:51.345 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:35:51.383 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:35:51.399 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:35:51.399 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:51.399 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:35:51.399 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:52.521 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:52.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-03-29T10:35:53.412 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:46 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:53.561 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:53.562 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-03-29T10:35:53.563 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:53.781 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 583 kB/s | 127 kB 00:00 2024-03-29T10:35:53.781 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:53.781 INFO:teuthology.orchestra.run.smithi151.stdout:Total 579 kB/s | 127 kB 00:00 2024-03-29T10:35:53.782 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:53.787 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:53.787 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:53.843 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:53.843 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:53.986 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:54.105 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.827 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:54.828 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:54.828 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:54.828 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:55.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-03-29T10:35:55.647 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:48 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:55.750 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:55.804 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:55.805 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:35:55.806 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:55.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-03-29T10:35:56.445 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:49 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:56.598 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:56.598 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:56.598 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:35:56.599 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:56.600 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 31 k 2024-03-29T10:35:56.600 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 140 k 2024-03-29T10:35:56.600 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:35:57.078 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 65 kB/s | 31 kB 00:00 2024-03-29T10:35:57.079 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:57.079 INFO:teuthology.orchestra.run.smithi151.stdout:Total 65 kB/s | 31 kB 00:00 2024-03-29T10:35:57.079 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:35:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:35:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:35:57.117 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:35:57.117 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:35:57.240 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:35:57.344 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:57.738 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:58.018 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:58.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:58.018 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:35:58.018 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:58.018 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:35:58.019 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:58.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-03-29T10:35:58.749 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:51 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:58.847 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:58.898 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:58.899 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:35:58.899 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:58.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-03-29T10:35:59.510 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:52 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:35:59.609 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:59.663 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:35:59.664 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:35:59.664 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:35:59.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-03-29T10:36:00.270 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:53 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:00.369 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:00.421 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:00.422 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:36:00.422 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:00.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-03-29T10:36:01.033 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:54 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:01.133 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:01.185 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:01.186 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:36:01.186 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:01.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-03-29T10:36:01.816 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:54 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:01.922 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:01.978 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:01.978 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:36:01.979 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:02.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-03-29T10:36:02.627 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:55 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:02.730 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:02.785 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:02.786 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-03-29T10:36:02.786 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:02.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-03-29T10:36:03.403 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:56 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:03.550 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:03.551 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:03.551 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-29T10:36:03.551 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-03-29T10:36:03.552 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:36:03.916 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 234 kB/s | 85 kB 00:00 2024-03-29T10:36:03.917 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:03.917 INFO:teuthology.orchestra.run.smithi151.stdout:Total 233 kB/s | 85 kB 00:00 2024-03-29T10:36:03.917 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:36:03.924 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:36:03.924 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:36:03.980 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:36:03.981 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:36:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:36:04.231 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:04.656 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:04.959 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:05.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-03-29T10:36:05.812 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:03:58 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:05.969 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:36:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-03-29T10:36:05.971 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:36:06.381 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.5 MB/s | 3.1 MB 00:00 2024-03-29T10:36:06.382 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:06.382 INFO:teuthology.orchestra.run.smithi151.stdout:Total 7.5 MB/s | 3.1 MB 00:00 2024-03-29T10:36:06.382 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:36:06.391 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:36:06.391 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:36:06.464 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:36:06.465 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:36:06.897 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:36:06.957 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:36:06.982 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:08.134 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:08.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-03-29T10:36:08.963 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:04:01 ago on Fri 29 Mar 2024 10:32:07 AM UTC. 2024-03-29T10:36:09.113 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-29T10:36:09.113 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:09.113 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-29T10:36:09.113 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-29T10:36:09.114 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-29T10:36:09.115 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:09.115 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 171 k 2024-03-29T10:36:09.115 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-03-29T10:36:09.115 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-29T10:36:09.655 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 317 kB/s | 171 kB 00:00 2024-03-29T10:36:09.655 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:09.656 INFO:teuthology.orchestra.run.smithi151.stdout:Total 316 kB/s | 171 kB 00:00 2024-03-29T10:36:09.656 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-29T10:36:09.664 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-29T10:36:09.664 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-29T10:36:09.721 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-29T10:36:09.721 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-29T10:36:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-29T10:36:09.988 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:10.426 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:10.725 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-29T10:36:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:10.726 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-29T10:36:10.959 DEBUG:teuthology.parallel:result is None 2024-03-29T10:36:10.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:36:11.105 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:36:11.125 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:36:11.126 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:36:11.126 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:36:11.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:36:11.515 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:36:11.540 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:36:11.540 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:36:11.540 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:36:11.542 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:36:11.542 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:11.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:36:11.570 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:11.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:36:11.611 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:36:11.611 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:11.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:36:11.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:36:11.705 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:11.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:36:11.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:36:11.805 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:36:11.806 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:11.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:36:11.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:36:11.899 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:11.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:36:11.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:36:11.998 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:36:11.999 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:11.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:36:12.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:36:12.092 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:12.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:36:12.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:36:12.186 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:36:12.275 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:36:12.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2024-03-29T10:36:12.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-03-29T10:36:12.306 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:36:12.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:36:12.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:36:12.373 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:36:12.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:36:12.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:36:12.474 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:36:12.474 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:36:12.474 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622638', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBx3xnHFbtJ25zGMuZ04L5GpSugqYgVK3jBBkv1e9NwE33Pz/SKbJjoF2qr6FsGun4tBX1imiEFrm8gjCWJMj7Q=', 'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBY17LjcqhnrYdAm1fZk6fnGHYKtVuScVWQbsPTF73sszU4IAltJPgULm3CjqaD+OCMB9+GTv1bb7/2nO75Q508='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:36:12.475 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:36:12.480 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:36:12.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:36:12.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:36:12.562 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:12.562 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:36:12.615 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:36:12.615 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-29 10:36:07.245113911 +0000 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-29 10:31:17.954551035 +0000 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-29 10:31:17.954551035 +0000 2024-03-29T10:36:12.670 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-29 10:31:17.954551035 +0000 2024-03-29T10:36:12.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:36:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-29T10:36:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-29T10:36:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000204643 s, 2.5 MB/s 2024-03-29T10:36:12.737 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:36:12.793 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-29T10:36:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:36:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:12.847 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:36:12.847 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:12.847 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:12.848 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-29 10:36:07.246113900 +0000 2024-03-29T10:36:12.848 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-29 10:31:18.307547359 +0000 2024-03-29T10:36:12.848 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-29 10:31:18.307547359 +0000 2024-03-29T10:36:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-29 10:31:18.307547359 +0000 2024-03-29T10:36:12.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:36:12.912 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-29T10:36:12.913 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-29T10:36:12.913 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000202582 s, 2.5 MB/s 2024-03-29T10:36:12.914 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:36:12.969 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-29T10:36:13.024 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-29 10:36:07.246113900 +0000 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-29 10:31:18.657543714 +0000 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-29 10:31:18.657543714 +0000 2024-03-29T10:36:13.025 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-29 10:31:18.657543714 +0000 2024-03-29T10:36:13.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:36:13.090 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-29T10:36:13.090 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-29T10:36:13.090 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000297428 s, 1.7 MB/s 2024-03-29T10:36:13.091 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:36:13.147 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-29 10:36:07.247113889 +0000 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-29 10:31:18.998540163 +0000 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-29 10:31:18.998540163 +0000 2024-03-29T10:36:13.203 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-29 10:31:18.998540163 +0000 2024-03-29T10:36:13.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:36:13.269 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-29T10:36:13.269 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-29T10:36:13.269 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000242207 s, 2.1 MB/s 2024-03-29T10:36:13.270 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:36:13.326 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:13.326 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:13.326 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:36:13.341 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:36:13.341 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-29 10:34:01.892261565 +0000 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-29 10:31:18.233447300 +0000 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-29 10:31:18.233447300 +0000 2024-03-29T10:36:13.397 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-29 10:31:18.233447300 +0000 2024-03-29T10:36:13.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:36:13.463 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-29T10:36:13.463 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-29T10:36:13.463 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000262063 s, 2.0 MB/s 2024-03-29T10:36:13.464 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:36:13.521 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-29 10:34:01.893261544 +0000 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-29 10:31:18.576440108 +0000 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-29 10:31:18.576440108 +0000 2024-03-29T10:36:13.576 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-29 10:31:18.576440108 +0000 2024-03-29T10:36:13.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:36:13.641 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-29T10:36:13.642 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-29T10:36:13.642 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000207038 s, 2.5 MB/s 2024-03-29T10:36:13.643 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:36:13.698 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-29 10:34:01.893261544 +0000 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-29 10:31:18.915433000 +0000 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-29 10:31:18.915433000 +0000 2024-03-29T10:36:13.754 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-29 10:31:18.915433000 +0000 2024-03-29T10:36:13.755 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:36:13.819 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-29T10:36:13.819 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-29T10:36:13.819 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000213476 s, 2.4 MB/s 2024-03-29T10:36:13.820 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:36:13.875 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-29T10:36:13.929 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:36:13.929 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-29 10:34:01.893261544 +0000 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-29 10:31:19.267425619 +0000 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-29 10:31:19.267425619 +0000 2024-03-29T10:36:13.930 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-29 10:31:19.267425619 +0000 2024-03-29T10:36:13.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:36:13.995 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-29T10:36:13.995 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-29T10:36:13.995 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000195705 s, 2.6 MB/s 2024-03-29T10:36:13.996 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:36:14.052 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:14.052 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:36:14.053 INFO:tasks.ceph:Generating config... 2024-03-29T10:36:14.054 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:36:14.054 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:36:14.054 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:36:14.054 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:36:14.054 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:36:14.055 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:36:14.055 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:36:14.055 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:36:14.056 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:36:14.057 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:36:14.057 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:36:14.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:36:14.116 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:36:14.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:36:14.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:36:14.192 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.151:6789]'), ('mon.c', '[v1:172.21.15.151:6790]'), ('mon.b', '[v1:172.21.15.169:6789]')] 2024-03-29T10:36:14.192 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.151:6789],[v1:172.21.15.151:6790],[v1:172.21.15.169:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:36:14.193 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:14.193 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:36:14.248 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.151:6789]' --addv c '[v1:172.21.15.151:6790]' --addv b '[v1:172.21.15.169:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: generated fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = quincy 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 0 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:min_mon_release 17 (quincy) 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:election_strategy: 1 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:14.335 INFO:teuthology.orchestra.run.smithi151.stdout:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:14.336 INFO:teuthology.orchestra.run.smithi151.stdout:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:14.336 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:36:14.337 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:36:14.352 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e... 2024-03-29T10:36:14.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:36:14.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:36:14.443 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-03-29T10:36:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:36:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:36:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "[v1:172.21.15.151:6789],[v1:172.21.15.151:6790],[v1:172.21.15.169:6789]" 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: lockdep = True 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon client directed command retry = 5 2024-03-29T10:36:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: mon election default strategy = 1 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: ms bind msgr2 = False 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: ms inject socket failures = 5000 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: ms type = async 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-03-29T10:36:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug trim objects = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-03-29T10:36:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore compression algorithm = zstd 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore compression mode = aggressive 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore zero block detection = True 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 20 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 20 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-29T10:36:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd class default list = * 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd class load list = * 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd client watch timeout = 120 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify cached snaps = True 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify missing on start = True 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock override recovery settings = True 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue = debug_random 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd op queue cut off = debug_random 2024-03-29T10:36:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: osd shutdown pgref assert = True 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-29T10:36:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:36:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon min osdmap epochs = 50 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune interval = 2 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune min = 15 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon scrub interval = 300 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on pool no app = False 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: paxos service trim min = 10 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:14.453 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: debug monc = 20 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: debug objecter = 20 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: debug rados = 20 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2024-03-29T10:36:14.454 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-29T10:36:14.455 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-29T10:36:14.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-29T10:36:14.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "[v1:172.21.15.151:6789],[v1:172.21.15.151:6790],[v1:172.21.15.169:6789]" 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: lockdep = True 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: mon client directed command retry = 5 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: mon election default strategy = 1 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: ms bind msgr2 = False 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject socket failures = 5000 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: ms type = async 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-29T10:36:14.458 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-29T10:36:14.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-29T10:36:14.460 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression algorithm = zstd 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression mode = aggressive 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore zero block detection = True 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 20 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 20 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 10 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: osd class default list = * 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: osd class load list = * 2024-03-29T10:36:14.461 INFO:teuthology.orchestra.run.smithi169.stdout: osd client watch timeout = 120 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify cached snaps = True 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify missing on start = True 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock override recovery settings = True 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue = debug_random 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue cut off = debug_random 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-29T10:36:14.462 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-29T10:36:14.463 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.464 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon min osdmap epochs = 50 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune interval = 2 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune min = 15 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon scrub interval = 300 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on pool no app = False 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: paxos service trim min = 10 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-29T10:36:14.465 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-29T10:36:14.466 INFO:teuthology.orchestra.run.smithi169.stdout: debug objecter = 20 2024-03-29T10:36:14.466 INFO:teuthology.orchestra.run.smithi169.stdout: debug rados = 20 2024-03-29T10:36:14.466 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-29T10:36:14.466 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-29T10:36:14.466 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-29T10:36:14.467 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:36:14.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:36:14.563 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:36:14.563 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:14.563 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:36:14.578 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:14.578 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:36:14.633 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:36:14.633 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:14.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:36:14.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:36:14.712 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:14.712 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:14.767 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:36:14.767 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:14.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:36:14.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:36:14.806 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:14.807 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:14.861 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:36:14.862 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:36:14.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:36:14.916 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:36:14.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:36:14.989 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:36:14.991 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:36:14.991 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:36:14.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:36:15.033 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:36:15.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:36:15.090 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:36:15.105 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:36:15.105 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:36:15.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:36:15.132 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:15.132 INFO:tasks.ceph:role: osd.0 2024-03-29T10:36:15.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:36:15.132 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:36:15.206 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:15.206 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:15.206 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:15.207 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:15.707 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-29T10:36:15.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-29T10:36:15.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:36:15.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:36:15.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:36:15.943 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:15.943 INFO:tasks.ceph:role: osd.1 2024-03-29T10:36:15.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:36:15.943 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:16.017 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-29T10:36:16.470 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-29T10:36:16.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:36:16.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:36:16.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:36:16.667 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:16.667 INFO:tasks.ceph:role: osd.2 2024-03-29T10:36:16.667 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:36:16.667 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:36:16.742 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:16.743 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:17.232 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-29T10:36:17.234 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-29T10:36:17.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:36:17.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:36:17.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:36:17.483 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:17.483 INFO:tasks.ceph:role: osd.3 2024-03-29T10:36:17.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:36:17.483 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:17.558 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:18.071 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-29T10:36:18.073 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-29T10:36:18.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:36:18.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:36:18.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:18.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.370+0000 7f3a3bfe1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:36:18.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.370+0000 7f3a3bfe1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:36:18.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.370+0000 7f3a3bfe1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:36:18.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.370+0000 7f3a3bfe1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:36:18.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.370+0000 7f3a3bfe1640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:36:18.357 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:18.371+0000 7f3a3bfe1640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:36:21.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:36:21.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:21.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.338+0000 7fb04d51f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:36:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.338+0000 7fb04d51f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:36:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.338+0000 7fb04d51f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:36:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.338+0000 7fb04d51f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:36:21.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.338+0000 7fb04d51f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:36:21.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:21.339+0000 7fb04d51f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:36:24.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:36:24.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.303+0000 7ff6b2e1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.303+0000 7ff6b2e1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.303+0000 7ff6b2e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.303+0000 7ff6b2e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.303+0000 7ff6b2e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:36:24.289 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:24.304+0000 7ff6b2e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:36:27.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:36:27.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:27.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.282+0000 7f9157793640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:36:27.267 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.283+0000 7f9157793640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:36:27.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.283+0000 7f9157793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:36:27.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.283+0000 7f9157793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:36:27.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.283+0000 7f9157793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:36:27.268 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:27.283+0000 7f9157793640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:36:30.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:36:30.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:36:30.181 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:30.181 INFO:tasks.ceph:role: osd.4 2024-03-29T10:36:30.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:36:30.181 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:36:30.260 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:30.260 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:30.261 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:30.775 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-29T10:36:30.777 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-29T10:36:30.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:36:30.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:36:30.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:36:31.038 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:31.038 INFO:tasks.ceph:role: osd.5 2024-03-29T10:36:31.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:36:31.039 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:31.113 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:31.114 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:31.114 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:31.114 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:31.114 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:31.568 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-29T10:36:31.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-29T10:36:31.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:36:31.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:36:31.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:36:31.780 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:31.780 INFO:tasks.ceph:role: osd.6 2024-03-29T10:36:31.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:36:31.781 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:36:31.856 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:31.856 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:31.856 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:31.856 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:31.857 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:32.353 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-29T10:36:32.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-29T10:36:32.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:36:32.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:36:32.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:36:32.584 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:36:32.585 INFO:tasks.ceph:role: osd.7 2024-03-29T10:36:32.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:36:32.585 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:36:32.660 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:36:32.661 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:36:33.161 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-29T10:36:33.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-29T10:36:33.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:36:33.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:36:33.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:33.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.460+0000 7f797da10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:36:33.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.461+0000 7f797da10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:36:33.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.461+0000 7f797da10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:36:33.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.461+0000 7f797da10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:36:33.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.461+0000 7f797da10640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:36:33.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:33.461+0000 7f797da10640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:36:36.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:36:36.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:36.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.459+0000 7fe03fe99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:36:36.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.460+0000 7fe03fe99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:36:36.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.460+0000 7fe03fe99640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:36:36.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.460+0000 7fe03fe99640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:36:36.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.460+0000 7fe03fe99640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:36:36.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:36.460+0000 7fe03fe99640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:36:39.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:36:39.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.434+0000 7fd6c0e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.434+0000 7fd6c0e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.434+0000 7fd6c0e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.434+0000 7fd6c0e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.434+0000 7fd6c0e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:36:39.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:39.435+0000 7fd6c0e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:36:42.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:36:42.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.410+0000 7f3c31419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.410+0000 7f3c31419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.410+0000 7f3c31419640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.410+0000 7f3c31419640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.410+0000 7f3c31419640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:36:42.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:42.411+0000 7f3c31419640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:36:45.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:36:45.271 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:36:45.271 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:36:45.304 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:36:45.375 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:36:45.445 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:36:45.514 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:36:45.583 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:36:45.614 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:36:45.686 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:36:45.756 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:36:45.826 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:36:45.895 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:45.896 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:36:45.912 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:45.912 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:36:45.951 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:36:45.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:36:45.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCemQZmoKmXNxAAev6RtH6uZxpldT/3IWrWAg== 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCimQZm0tAxFhAA0louvKs9BAhOEwnA2Iv5EA== 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-03-29T10:36:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQClmQZma4RIFBAA2xD1cSFV+O+NF3iO9GDxIQ== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQComQZmZVAyEhAAwZQ9a9LF4g2O7nRRvXShHQ== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCrmQZmOB3pEBAA6UhWalobyR2vmWrVKZzy8Q== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCfmQZmZNNTABAAZWIJwSTZBYPm6K4ZEZDH1w== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCxmQZmLkp+GxAAjD+VxkbXi62zcro0cuoJcA== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQC0mQZmk1RyGxAADjfpm1x3JGeIsa/DIbdNbQ== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQC3mQZmzjD8GRAAIYkoOY8TK7dayy14oaXbGg== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQC6mQZmCdmQGBAA7h6mwx400N1WX0o1sselXg== 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-03-29T10:36:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCfmQZmIhv7AhAAFKMsgCENVxqxT/Oggbwjdg== 2024-03-29T10:36:45.996 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-03-29T10:36:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCfmQZmCvpgBhAAR5qrv5lEdB+NkJ+mgdKQXg== 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.y] 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCemQZmoKmXNxAAev6RtH6uZxpldT/3IWrWAg== 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCimQZm0tAxFhAA0louvKs9BAhOEwnA2Iv5EA== 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQClmQZma4RIFBAA2xD1cSFV+O+NF3iO9GDxIQ== 2024-03-29T10:36:46.017 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQComQZmZVAyEhAAwZQ9a9LF4g2O7nRRvXShHQ== 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.3] 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCrmQZmOB3pEBAA6UhWalobyR2vmWrVKZzy8Q== 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCfmQZmZNNTABAAZWIJwSTZBYPm6K4ZEZDH1w== 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.4] 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCxmQZmLkp+GxAAjD+VxkbXi62zcro0cuoJcA== 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.5] 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQC0mQZmk1RyGxAADjfpm1x3JGeIsa/DIbdNbQ== 2024-03-29T10:36:46.018 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.6] 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQC3mQZmzjD8GRAAIYkoOY8TK7dayy14oaXbGg== 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.7] 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQC6mQZmCdmQGBAA7h6mwx400N1WX0o1sselXg== 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCfmQZmIhv7AhAAFKMsgCENVxqxT/Oggbwjdg== 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2024-03-29T10:36:46.019 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCfmQZmCvpgBhAAR5qrv5lEdB+NkJ+mgdKQXg== 2024-03-29T10:36:46.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:36:46.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:36:46.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:36:46.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:36:46.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:36:46.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:36:46.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:36:46.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:36:46.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:36:46.720 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:36:46.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:36:46.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:36:46.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:36:46.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:36:46.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:36:47.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:36:47.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:36:47.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:36:47.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:36:47.214 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:47.217 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:36:47.269 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:36:47.270 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:36:47.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:36:47.272 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:36:47.272 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:36:47.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:36:47.274 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:36:47.274 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:36:47.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:36:47.312 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:36:47.312 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:36:47.313 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:36:47.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:36:47.315 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:36:47.315 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:36:47.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:36:47.317 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:36:47.317 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:36:47.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:36:47.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.433+0000 7f7f1959d640 10 monclient: get_monmap_and_config 2024-03-29T10:36:47.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.433+0000 7f7f1959d640 10 monclient: build_initial_monmap 2024-03-29T10:36:47.418 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.433+0000 7f7f1959d640 10 monclient: monmap: 2024-03-29T10:36:47.418 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:47.434873+0000 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:47.434873+0000 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:47.419 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.434+0000 7f7f1959d640 1 Processor -- start 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.434+0000 7f7f1959d640 1 -- start start 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.434+0000 7f7f1959d640 10 monclient: init 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f1409ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f1409c870 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f140b4360 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1409dec0 con 0x7f7f1409c870 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1409da30 con 0x7f7f1409ac40 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f137fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f7f140b4360 0x7f7f1414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:50442/0 (socket says 172.21.15.151:50442) 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f137fe640 1 -- 172.21.15.151:0/2988933414 learned_addr learned my addr 172.21.15.151:0/2988933414 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b39c0 con 0x7f7f140b4360 2024-03-29T10:36:47.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:47.422 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:47.435+0000 7f7f1959d640 10 monclient(hunting): authenticate will time out at 938.605238s 2024-03-29T10:36:50.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): tick 2024-03-29T10:36:50.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:36:50.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 monclient(hunting): continuing hunt 2024-03-29T10:36:50.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:36:50.420 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.151:6790/0 conn(0x7f7f140b4360 legacy=0x7f7f1414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.169:6789/0 conn(0x7f7f1409ac40 legacy=0x7f7f140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.151:6789/0 conn(0x7f7f1409c870 legacy=0x7f7f1413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef40036e0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef40072d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef400b020 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b5160 con 0x7f7ef400b020 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef400fdf0 con 0x7f7ef40036e0 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef4011050 con 0x7f7ef40072d0 2024-03-29T10:36:50.421 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:50.435+0000 7f7f117fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.604+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512049105 0 0) 0x7f7f140b5160 con 0x7f7ef400b020 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:36:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f00003520 con 0x7f7ef400b020 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269081452 0 0) 0x7f7ef4011050 con 0x7f7ef40072d0 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f140b5160 con 0x7f7ef40072d0 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531597330 0 0) 0x7f7ef400fdf0 con 0x7f7ef40036e0 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef4011050 con 0x7f7ef40036e0 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310436762 0 0) 0x7f7f140b5160 con 0x7f7ef40072d0 2024-03-29T10:36:52.591 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.169:6789/0 conn(0x7f7ef40036e0 legacy=0x7f7ef4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.605+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.151:6789/0 conn(0x7f7ef400b020 legacy=0x7f7ef400d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 1 monclient: found mon.noname-b 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4012290 con 0x7f7ef40072d0 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7efc001470 con 0x7f7ef40072d0 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: dump: 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:52.592 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f1959d640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:36:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f1959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:52.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7efc0015d0 con 0x7f7ef40072d0 2024-03-29T10:36:52.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:52.598 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 1 -- 172.21.15.151:0/2988933414 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7efc0019e0 con 0x7f7ef40072d0 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f127fc640 10 monclient: dump: 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:52.599 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f1959d640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.606+0000 7f7f1959d640 4 set_mon_vals no callback set 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 10 monclient: shutdown 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 >> v1:172.21.15.151:6790/0 conn(0x7f7ef40072d0 legacy=0x7f7ef4009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 shutdown_connections 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 >> 172.21.15.151:0/2988933414 conn(0x7f7f14066b00 msgr2=0x7f7f14066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 shutdown_connections 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.607+0000 7f7f1959d640 1 -- 172.21.15.151:0/2988933414 wait complete. 2024-03-29T10:36:52.600 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.608+0000 7f7f1959d640 10 monclient: build_initial_monmap 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.608+0000 7f7f1959d640 10 monclient: monmap: 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.608+0000 7f7f1959d640 1 librados: starting msgr at 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.608+0000 7f7f1959d640 1 librados: starting objecter 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.608+0000 7f7f1959d640 1 Processor -- start 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 1 -- start start 2024-03-29T10:36:52.601 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 1 librados: setting wanted keys 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 1 librados: calling monclient init 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 10 monclient: init 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef40036e0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.609+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef40072d0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f12ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f7ef40036e0 0x7f7f140b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60928/0 (socket says 172.21.15.151:60928) 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f7ef40072d0 0x7f7f141d7270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:57484/0 (socket says 172.21.15.151:57484) 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef400b020 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f12ffd640 1 -- 172.21.15.151:0/1705520593 learned_addr learned my addr 172.21.15.151:0/1705520593 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f141d6de0 con 0x7f7ef400b020 2024-03-29T10:36:52.602 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7ef40036e0 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b6ef0 con 0x7f7ef40072d0 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140b71d0 con 0x7f7ef40036e0 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7ef40072d0 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter ms_handle_connect 0x7f7ef400b020 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f1959d640 10 monclient(hunting): authenticate will time out at 943.780171s 2024-03-29T10:36:52.603 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722550651 0 0) 0x7f7f141d6de0 con 0x7f7ef400b020 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.610+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef40119a0 con 0x7f7ef400b020 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993298862 0 0) 0x7f7ef40119a0 con 0x7f7ef400b020 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f141d6de0 con 0x7f7ef400b020 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7f0c001bf0 con 0x7f7ef400b020 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147302089 0 0) 0x7f7f141d6de0 con 0x7f7ef400b020 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 >> v1:172.21.15.151:6790/0 conn(0x7f7ef40036e0 legacy=0x7f7f140b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 >> v1:172.21.15.169:6789/0 conn(0x7f7ef40072d0 legacy=0x7f7f141d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 monclient: found mon.a 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:52.605 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140b75f0 con 0x7f7ef400b020 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 10 monclient: _renew_subs 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f141d7980 con 0x7f7ef400b020 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 10 monclient: _renew_subs 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f141d7cd0 con 0x7f7ef400b020 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f1959d640 1 librados: init done 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f0c0020e0 con 0x7f7ef400b020 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7f0c004d00 con 0x7f7ef400b020 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f11ffb640 4 set_mon_vals no callback set 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.611+0000 7f7f0b7fe640 10 monclient: dump: 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.607 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7f0c0052b0 con 0x7f7ef400b020 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 419352051 0 0) 0x7f7f0c006830 con 0x7f7ef400b020 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 10 client.4113.objecter ms_dispatch 0x7f7f14000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.612+0000 7f7f0b7fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.613+0000 7f7f1959d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.613+0000 7f7f1959d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.613+0000 7f7f1959d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.613+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef8005080 con 0x7f7ef400b020 2024-03-29T10:36:52.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.616+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7f0c006ad0 con 0x7f7ef400b020 2024-03-29T10:36:52.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.616+0000 7f7f0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.609 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.616+0000 7f7f0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:52.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.683+0000 7f7f1959d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:36:52.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.683+0000 7f7f1959d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:36:52.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.683+0000 7f7f1959d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:52.668 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.683+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7ef8005330 con 0x7f7ef400b020 2024-03-29T10:36:52.669 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.685+0000 7f7f0b7fe640 1 -- 172.21.15.151:0/1705520593 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f7f0c005790 con 0x7f7ef400b020 2024-03-29T10:36:52.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.685+0000 7f7f0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:36:52.670 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.685+0000 7f7f0b7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:36:52.670 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-03-29T10:36:52.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 10 librados: watch_flush enter 2024-03-29T10:36:52.671 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 10 librados: watch_flush exit 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 10 monclient: shutdown 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 >> v1:172.21.15.151:6789/0 conn(0x7f7ef400b020 legacy=0x7f7f141e1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 shutdown_connections 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 >> 172.21.15.151:0/1705520593 conn(0x7f7f14066b00 msgr2=0x7f7f141456e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 shutdown_connections 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 -- 172.21.15.151:0/1705520593 wait complete. 2024-03-29T10:36:52.672 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:52.686+0000 7f7f1959d640 1 librados: shutdown 2024-03-29T10:36:52.682 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:36:52.682 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:36:52.683 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:52.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:36:52.710 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:52.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:36:52.777 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:52.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:36:52.845 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-29T10:36:52.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:36:52.913 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:52.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:36:52.941 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:52.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:36:53.009 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:53.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:36:53.066 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.081+0000 7fd297238040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:36:53.070 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.086+0000 7f31d2e45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:36:53.076 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-29T10:36:53.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:36:53.122 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.137+0000 7fd297238040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:36:53.124 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.140+0000 7f31d2e45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:36:53.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new cf044ce8-1259-4cd0-a749-9de9ff157703 0 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.300+0000 7f12e513d640 10 monclient: get_monmap_and_config 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.300+0000 7f12e513d640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.300+0000 7f12e513d640 10 monclient: monmap: 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:53.302056+0000 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:53.302056+0000 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.286 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 1 Processor -- start 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 1 -- start start 2024-03-29T10:36:53.287 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient: init 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-a con 0x7f12e009ac40 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-b con 0x7f12e009c870 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-c con 0x7f12e00b4360 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e009dec0 con 0x7f12e009ac40 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e009da30 con 0x7f12e00b4360 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00b39c0 con 0x7f12e009c870 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12e513d640 10 monclient(hunting): authenticate will time out at 941.587595s 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12df59e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f12e00b4360 0x7f12e014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50390/0 (socket says 172.21.15.169:50390) 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.301+0000 7f12df59e640 1 -- 172.21.15.169:0/4291539525 learned_addr learned my addr 172.21.15.169:0/4291539525 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667618148 0 0) 0x7f12e009da30 con 0x7f12e00b4360 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12cc003520 con 0x7f12e00b4360 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657705350 0 0) 0x7f12d0001490 con 0x7f12e009ac40 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e009da30 con 0x7f12e009ac40 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027840525 0 0) 0x7f12e00b39c0 con 0x7f12e009c870 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d0001490 con 0x7f12e009c870 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660430030 0 0) 0x7f12cc003520 con 0x7f12e00b4360 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 >> v1:172.21.15.151:6790/0 conn(0x7f12e009c870 legacy=0x7f12e013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 >> v1:172.21.15.151:6789/0 conn(0x7f12e009ac40 legacy=0x7f12e00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 monclient: found mon.noname-c 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00b5160 con 0x7f12e00b4360 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12d8001690 con 0x7f12e00b4360 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12e513d640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12e513d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: dump: 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12e513d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d80017f0 con 0x7f12e00b4360 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 1 -- 172.21.15.169:0/4291539525 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12d8001c00 con 0x7f12e00b4360 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12e513d640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12e513d640 4 set_mon_vals no callback set 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.302+0000 7f12ddd9b640 10 monclient: dump: 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 10 monclient: shutdown 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 1 -- 172.21.15.169:0/4291539525 >> v1:172.21.15.169:6789/0 conn(0x7f12e00b4360 legacy=0x7f12e014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 1 -- 172.21.15.169:0/4291539525 shutdown_connections 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 1 -- 172.21.15.169:0/4291539525 >> 172.21.15.169:0/4291539525 conn(0x7f12e0066b00 msgr2=0x7f12e0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 1 -- 172.21.15.169:0/4291539525 shutdown_connections 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.303+0000 7f12e513d640 1 -- 172.21.15.169:0/4291539525 wait complete. 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.304+0000 7f12e513d640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.304+0000 7f12e513d640 10 monclient: monmap: 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:53.301 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.304+0000 7f12e513d640 1 librados: starting msgr at 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.304+0000 7f12e513d640 1 librados: starting objecter 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 1 Processor -- start 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 1 -- start start 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 1 librados: setting wanted keys 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 1 librados: calling monclient init 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient: init 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-b con 0x7f12e009ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-c con 0x7f12e01443b0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 10 monclient(hunting): picked mon.noname-a con 0x7f12e0140920 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.305+0000 7f12e513d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0142d80 con 0x7f12e0140920 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e01da7b0 con 0x7f12e009ac40 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0141bd0 con 0x7f12e01443b0 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient(hunting): authenticate will time out at 941.591919s 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12ded9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f12e009ac40 0x7f12e01414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50404/0 (socket says 172.21.15.169:50404) 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12ded9d640 1 -- 172.21.15.169:0/453350124 learned_addr learned my addr 172.21.15.169:0/453350124 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter ms_handle_connect 0x7f12e009ac40 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter ms_handle_connect 0x7f12e0140920 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter ms_handle_connect 0x7f12e01443b0 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490642229 0 0) 0x7f12e01da7b0 con 0x7f12e009ac40 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0004eb0 con 0x7f12e009ac40 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689101468 0 0) 0x7f12e0141bd0 con 0x7f12e01443b0 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e01da7b0 con 0x7f12e01443b0 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059713173 0 0) 0x7f12e0142d80 con 0x7f12e0140920 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e0141bd0 con 0x7f12e0140920 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040258906 0 0) 0x7f12c0004eb0 con 0x7f12e009ac40 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12e0142d80 con 0x7f12e009ac40 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12d0001da0 con 0x7f12e009ac40 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.305 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841832780 0 0) 0x7f12e01da7b0 con 0x7f12e01443b0 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12c0004eb0 con 0x7f12e01443b0 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12c8001870 con 0x7f12e01443b0 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.306 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3256944917 0 0) 0x7f12e0141bd0 con 0x7f12e0140920 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12e01da7b0 con 0x7f12e0140920 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12d8001960 con 0x7f12e0140920 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.308 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3299522778 0 0) 0x7f12e0142d80 con 0x7f12e009ac40 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 >> v1:172.21.15.151:6790/0 conn(0x7f12e01443b0 legacy=0x7f12e01dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 >> v1:172.21.15.151:6789/0 conn(0x7f12e0140920 legacy=0x7f12e0136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 monclient: found mon.b 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0141d60 con 0x7f12e009ac40 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient: _renew_subs 2024-03-29T10:36:53.309 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e01db3a0 con 0x7f12e009ac40 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient: _renew_subs 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e0137770 con 0x7f12e009ac40 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12e513d640 1 librados: init done 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12d0002060 con 0x7f12e009ac40 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.310 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12d0003de0 con 0x7f12e009ac40 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12dd59a640 4 set_mon_vals no callback set 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.306+0000 7f12d77fe640 10 monclient: dump: 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f12d0004390 con 0x7f12e009ac40 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2035976805 0 0) 0x7f12d0006a00 con 0x7f12e009ac40 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 10 client.4114.objecter ms_dispatch 0x7f12e0000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12d77fe640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12e513d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12e513d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12e513d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.307+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4005080 con 0x7f12e009ac40 2024-03-29T10:36:53.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.309+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f12d0005b40 con 0x7f12e009ac40 2024-03-29T10:36:53.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.309+0000 7f12d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.309+0000 7f12d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:53.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.380+0000 7f12e513d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cf044ce8-1259-4cd0-a749-9de9ff157703", "id": 0}] 2024-03-29T10:36:53.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.380+0000 7f12e513d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cf044ce8-1259-4cd0-a749-9de9ff157703", "id": 0}] 2024-03-29T10:36:53.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.380+0000 7f12e513d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.365 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.380+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cf044ce8-1259-4cd0-a749-9de9ff157703", "id": 0} v 0) v1 -- 0x7f12c40058b0 con 0x7f12e009ac40 2024-03-29T10:36:53.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12d77fe640 1 -- 172.21.15.169:0/453350124 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf044ce8-1259-4cd0-a749-9de9ff157703", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1974782008 0 732707515) 0x7f12d0005d60 con 0x7f12e009ac40 2024-03-29T10:36:53.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cf044ce8-1259-4cd0-a749-9de9ff157703", "id": 0}] 2024-03-29T10:36:53.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:53.371 INFO:teuthology.orchestra.run.smithi169.stdout:0 2024-03-29T10:36:53.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 10 librados: watch_flush enter 2024-03-29T10:36:53.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 10 librados: watch_flush exit 2024-03-29T10:36:53.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T10:36:53.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 10 client.4114.objecter successfully canceled tick 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 10 monclient: shutdown 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.386+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 >> v1:172.21.15.169:6789/0 conn(0x7f12e009ac40 legacy=0x7f12e01414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.387+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 shutdown_connections 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.387+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 >> 172.21.15.169:0/453350124 conn(0x7f12e0066b00 msgr2=0x7f12e013c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.387+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 shutdown_connections 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.387+0000 7f12e513d640 1 -- 172.21.15.169:0/453350124 wait complete. 2024-03-29T10:36:53.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.387+0000 7f12e513d640 1 librados: shutdown 2024-03-29T10:36:53.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 99db8471-c2b2-48af-93dc-e4b9b1de4bd3 1 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 10 monclient: get_monmap_and_config 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 10 monclient: monmap: 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:53.499803+0000 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:53.499803+0000 2024-03-29T10:36:53.483 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:53.484 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:53.484 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 1 Processor -- start 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 1 -- start start 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.498+0000 7fa2cd4a8640 10 monclient: init 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:53.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c80af360 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c806af60 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c8145e60 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c80b0520 con 0x7fa2c80af360 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c80aff20 con 0x7fa2c8145e60 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c80b1da0 con 0x7fa2c806af60 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2cd4a8640 10 monclient(hunting): authenticate will time out at 941.785356s 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa2c8145e60 0x7fa2c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50414/0 (socket says 172.21.15.169:50414) 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c77fe640 1 -- 172.21.15.169:0/2428912235 learned_addr learned my addr 172.21.15.169:0/2428912235 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157645278 0 0) 0x7fa2c80aff20 con 0x7fa2c8145e60 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b0003520 con 0x7fa2c8145e60 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853500489 0 0) 0x7fa2c80b1da0 con 0x7fa2c806af60 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c80aff20 con 0x7fa2c806af60 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576689259 0 0) 0x7fa2c80b0520 con 0x7fa2c80af360 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.499+0000 7fa2c5ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c80b1da0 con 0x7fa2c80af360 2024-03-29T10:36:53.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728639613 0 0) 0x7fa2b0003520 con 0x7fa2c8145e60 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 >> v1:172.21.15.151:6790/0 conn(0x7fa2c806af60 legacy=0x7fa2c8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 >> v1:172.21.15.151:6789/0 conn(0x7fa2c80af360 legacy=0x7fa2c806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 monclient: found mon.noname-c 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c80b2080 con 0x7fa2c8145e60 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2c0001a10 con 0x7fa2c8145e60 2024-03-29T10:36:53.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: dump: 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2c0001bf0 con 0x7fa2c8145e60 2024-03-29T10:36:53.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 1 -- 172.21.15.169:0/2428912235 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2c0001f80 con 0x7fa2c8145e60 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 4 set_mon_vals no callback set 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2c5ffb640 10 monclient: dump: 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.497 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 10 monclient: shutdown 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/2428912235 >> v1:172.21.15.169:6789/0 conn(0x7fa2c8145e60 legacy=0x7fa2c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/2428912235 shutdown_connections 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/2428912235 >> 172.21.15.169:0/2428912235 conn(0x7fa2c80b58f0 msgr2=0x7fa2c80b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/2428912235 shutdown_connections 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.500+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/2428912235 wait complete. 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.501+0000 7fa2cd4a8640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.501+0000 7fa2cd4a8640 10 monclient: monmap: 2024-03-29T10:36:53.498 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.501+0000 7fa2cd4a8640 1 librados: starting msgr at 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.501+0000 7fa2cd4a8640 1 librados: starting objecter 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.501+0000 7fa2cd4a8640 1 Processor -- start 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 -- start start 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 librados: setting wanted keys 2024-03-29T10:36:53.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 librados: calling monclient init 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient: init 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c806af60 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c80af360 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c8145e60 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c8121810 con 0x7fa2c8145e60 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c81dae60 con 0x7fa2c80af360 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c811f3b0 con 0x7fa2c806af60 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2cd4a8640 10 monclient(hunting): authenticate will time out at 941.788450s 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2c67fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa2c80af360 0x7fa2c81d6ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50430/0 (socket says 172.21.15.169:50430) 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2c67fc640 1 -- 172.21.15.169:0/3385622717 learned_addr learned my addr 172.21.15.169:0/3385622717 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter ms_handle_connect 0x7fa2c80af360 2024-03-29T10:36:53.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter ms_handle_connect 0x7fa2c8145e60 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter ms_handle_connect 0x7fa2c806af60 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693532148 0 0) 0x7fa2c81dae60 con 0x7fa2c80af360 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b4004eb0 con 0x7fa2c80af360 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683161263 0 0) 0x7fa2c8121810 con 0x7fa2c8145e60 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c81dae60 con 0x7fa2c8145e60 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573910386 0 0) 0x7fa2c811f3b0 con 0x7fa2c806af60 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.502+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c8121810 con 0x7fa2c806af60 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774393550 0 0) 0x7fa2b4004eb0 con 0x7fa2c80af360 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2c811f3b0 con 0x7fa2c80af360 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2bc001a30 con 0x7fa2c80af360 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.502 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2595122161 0 0) 0x7fa2c81dae60 con 0x7fa2c8145e60 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2b4004eb0 con 0x7fa2c8145e60 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2c0001f00 con 0x7fa2c8145e60 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408111940 0 0) 0x7fa2c8121810 con 0x7fa2c806af60 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2c81dae60 con 0x7fa2c806af60 2024-03-29T10:36:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2b8001760 con 0x7fa2c806af60 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2717491838 0 0) 0x7fa2c811f3b0 con 0x7fa2c80af360 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 >> v1:172.21.15.151:6790/0 conn(0x7fa2c806af60 legacy=0x7fa2c813e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 >> v1:172.21.15.151:6789/0 conn(0x7fa2c8145e60 legacy=0x7fa2c81e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 monclient: found mon.b 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8120810 con 0x7fa2c80af360 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 10 monclient: _renew_subs 2024-03-29T10:36:53.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c81da8e0 con 0x7fa2c80af360 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 10 monclient: _renew_subs 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2bc001cf0 con 0x7fa2c80af360 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c81dac30 con 0x7fa2c80af360 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2cd4a8640 1 librados: init done 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2bc003a60 con 0x7fa2c80af360 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 monclient: dump: 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.507 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2c57fa640 4 set_mon_vals no callback set 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa2bc003f50 con 0x7fa2c80af360 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1292400942 0 0) 0x7fa2bc006670 con 0x7fa2c80af360 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 10 client.4120.objecter ms_dispatch 0x7fa2c8000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.503+0000 7fa2af7fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T10:36:53.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.504+0000 7fa2cd4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.504+0000 7fa2cd4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.504+0000 7fa2cd4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.504+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c8067c00 con 0x7fa2c80af360 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.506+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa2bc004430 con 0x7fa2c80af360 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.506+0000 7fa2af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.506+0000 7fa2af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:53.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.579+0000 7fa2cd4a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "99db8471-c2b2-48af-93dc-e4b9b1de4bd3", "id": 1}] 2024-03-29T10:36:53.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.579+0000 7fa2cd4a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "99db8471-c2b2-48af-93dc-e4b9b1de4bd3", "id": 1}] 2024-03-29T10:36:53.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.579+0000 7fa2cd4a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.579+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "99db8471-c2b2-48af-93dc-e4b9b1de4bd3", "id": 1} v 0) v1 -- 0x7fa2c81daff0 con 0x7fa2c80af360 2024-03-29T10:36:53.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.587+0000 7fa2af7fe640 1 -- 172.21.15.169:0/3385622717 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "99db8471-c2b2-48af-93dc-e4b9b1de4bd3", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1910426725 0 940483276) 0x7fa2bc0058f0 con 0x7fa2c80af360 2024-03-29T10:36:53.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.587+0000 7fa2af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "99db8471-c2b2-48af-93dc-e4b9b1de4bd3", "id": 1}] 2024-03-29T10:36:53.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.587+0000 7fa2af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:53.573 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 10 librados: watch_flush enter 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 10 librados: watch_flush exit 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 10 client.4120.objecter successfully canceled tick 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 10 monclient: shutdown 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 >> v1:172.21.15.169:6789/0 conn(0x7fa2c80af360 legacy=0x7fa2c81d6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 shutdown_connections 2024-03-29T10:36:53.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 >> 172.21.15.169:0/3385622717 conn(0x7fa2c80b58f0 msgr2=0x7fa2c80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 shutdown_connections 2024-03-29T10:36:53.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 -- 172.21.15.169:0/3385622717 wait complete. 2024-03-29T10:36:53.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.588+0000 7fa2cd4a8640 1 librados: shutdown 2024-03-29T10:36:53.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 26b2e1b3-eb7a-4aad-9049-de761ad4fa50 2 2024-03-29T10:36:53.648 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.664+0000 7f31d2e45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.698+0000 7f17e124e640 10 monclient: get_monmap_and_config 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.698+0000 7f17e124e640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.698+0000 7f17e124e640 10 monclient: monmap: 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:53.700060+0000 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:53.700060+0000 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:53.684 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:53.685 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 1 Processor -- start 2024-03-29T10:36:53.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 1 -- start start 2024-03-29T10:36:53.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient: init 2024-03-29T10:36:53.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:53.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-c con 0x7f17dc09ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-a con 0x7f17dc09c870 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-b con 0x7f17dc0b4360 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc09dec0 con 0x7f17dc09c870 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc09da30 con 0x7f17dc09ac40 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0b39c0 con 0x7f17dc0b4360 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17e124e640 10 monclient(hunting): authenticate will time out at 941.985621s 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17dad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f17dc09ac40 0x7f17dc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50438/0 (socket says 172.21.15.169:50438) 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.699+0000 7f17dad9d640 1 -- 172.21.15.169:0/2845297218 learned_addr learned my addr 172.21.15.169:0/2845297218 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831803845 0 0) 0x7f17dc09da30 con 0x7f17dc09ac40 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c4003520 con 0x7f17dc09ac40 2024-03-29T10:36:53.690 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361903596 0 0) 0x7f17d0001110 con 0x7f17dc09c870 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc09da30 con 0x7f17dc09c870 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84095210 0 0) 0x7f17dc0b39c0 con 0x7f17dc0b4360 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d0001110 con 0x7f17dc0b4360 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976748468 0 0) 0x7f17c4003520 con 0x7f17dc09ac40 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 >> v1:172.21.15.151:6790/0 conn(0x7f17dc0b4360 legacy=0x7f17dc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 >> v1:172.21.15.151:6789/0 conn(0x7f17dc09c870 legacy=0x7f17dc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 monclient: found mon.noname-c 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0b5160 con 0x7f17dc09ac40 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17cc001690 con 0x7f17dc09ac40 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: dump: 2024-03-29T10:36:53.692 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17cc0017f0 con 0x7f17dc09ac40 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 1 -- 172.21.15.169:0/2845297218 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17cc001c00 con 0x7f17dc09ac40 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 4 set_mon_vals no callback set 2024-03-29T10:36:53.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17d9d9b640 10 monclient: dump: 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 10 monclient: shutdown 2024-03-29T10:36:53.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.700+0000 7f17e124e640 1 -- 172.21.15.169:0/2845297218 >> v1:172.21.15.169:6789/0 conn(0x7f17dc09ac40 legacy=0x7f17dc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 1 -- 172.21.15.169:0/2845297218 shutdown_connections 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 1 -- 172.21.15.169:0/2845297218 >> 172.21.15.169:0/2845297218 conn(0x7f17dc066b00 msgr2=0x7f17dc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 1 -- 172.21.15.169:0/2845297218 shutdown_connections 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 1 -- 172.21.15.169:0/2845297218 wait complete. 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.701+0000 7f17e124e640 10 monclient: monmap: 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.695 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 librados: starting msgr at 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 librados: starting objecter 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 Processor -- start 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 -- start start 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 librados: setting wanted keys 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 librados: calling monclient init 2024-03-29T10:36:53.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient: init 2024-03-29T10:36:53.696 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.702+0000 7fd297238040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:36:53.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:53.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-a con 0x7f17dc09ac40 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-c con 0x7f17dc0b5840 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 10 monclient(hunting): picked mon.noname-b con 0x7f17dc1cb7b0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.702+0000 7f17e124e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc1cbed0 con 0x7f17dc09ac40 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc136ce0 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0b7e70 con 0x7f17dc0b5840 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient(hunting): authenticate will time out at 941.988923s 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17dad9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f17dc09ac40 0x7f17dc1da4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.169:43652/0 (socket says 172.21.15.169:43652) 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17db59e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f17dc1cb7b0 0x7f17dc0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50454/0 (socket says 172.21.15.169:50454) 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17dad9d640 1 -- 172.21.15.169:0/4038784799 learned_addr learned my addr 172.21.15.169:0/4038784799 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter ms_handle_connect 0x7f17dc1cb7b0 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter ms_handle_connect 0x7f17dc09ac40 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter ms_handle_connect 0x7f17dc0b5840 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225654582 0 0) 0x7f17dc1cbed0 con 0x7f17dc09ac40 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c8004eb0 con 0x7f17dc09ac40 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45090019 0 0) 0x7f17dc0b7e70 con 0x7f17dc0b5840 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:36:53.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc1cbed0 con 0x7f17dc0b5840 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434593874 0 0) 0x7f17dc136ce0 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc0b7e70 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120314049 0 0) 0x7f17c8004eb0 con 0x7f17dc09ac40 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17dc136ce0 con 0x7f17dc09ac40 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17cc001b40 con 0x7f17dc09ac40 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.704 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3332890435 0 0) 0x7f17dc0b7e70 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17c8004eb0 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303924420 0 0) 0x7f17dc1cbed0 con 0x7f17dc0b5840 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17dc0b7e70 con 0x7f17dc0b5840 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17d4001b50 con 0x7f17dc1cb7b0 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.705 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17d00018a0 con 0x7f17dc0b5840 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:53.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient(hunting): dump: 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2728200655 0 0) 0x7f17dc136ce0 con 0x7f17dc09ac40 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 >> v1:172.21.15.151:6790/0 conn(0x7f17dc0b5840 legacy=0x7f17dc0b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 >> v1:172.21.15.169:6789/0 conn(0x7f17dc1cb7b0 legacy=0x7f17dc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 monclient: found mon.a 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc1cc430 con 0x7f17dc09ac40 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 5 monclient: authenticate success, global_id 4131 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient: _renew_subs 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc1dabe0 con 0x7f17dc09ac40 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient: _renew_subs 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:53.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc1db070 con 0x7f17dc09ac40 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17e124e640 1 librados: init done 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17cc004a90 con 0x7f17dc09ac40 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.703+0000 7f17c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f17cc001250 con 0x7f17dc09ac40 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17d959a640 4 set_mon_vals no callback set 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 10 monclient: dump: 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.709 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f17cc005230 con 0x7f17dc09ac40 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3013118777 0 0) 0x7f17cc0068d0 con 0x7f17dc09ac40 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 10 client.4131.objecter ms_dispatch 0x7f17dc000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17c37fe640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17e124e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17e124e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17e124e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.704+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc005080 con 0x7f17dc09ac40 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.707+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f17cc006560 con 0x7f17dc09ac40 2024-03-29T10:36:53.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.707+0000 7f17c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.707+0000 7f17c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:53.711 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.718+0000 7f31d2e45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:36:53.743 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.758+0000 7fd297238040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:36:53.747 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.763+0000 7f31d2e45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:36:53.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.778+0000 7f17e124e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26b2e1b3-eb7a-4aad-9049-de761ad4fa50", "id": 2}] 2024-03-29T10:36:53.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.778+0000 7f17e124e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26b2e1b3-eb7a-4aad-9049-de761ad4fa50", "id": 2}] 2024-03-29T10:36:53.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.778+0000 7f17e124e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:53.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.778+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd new", "uuid": "26b2e1b3-eb7a-4aad-9049-de761ad4fa50", "id": 2} v 0) v1 -- 0x7f17bc0058b0 con 0x7f17dc09ac40 2024-03-29T10:36:53.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.779+0000 7f17c37fe640 1 -- 172.21.15.169:0/4038784799 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26b2e1b3-eb7a-4aad-9049-de761ad4fa50", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1331578221 0 216599125) 0x7f17cc006b60 con 0x7f17dc09ac40 2024-03-29T10:36:53.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26b2e1b3-eb7a-4aad-9049-de761ad4fa50", "id": 2}] 2024-03-29T10:36:53.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:53.765 INFO:teuthology.orchestra.run.smithi169.stdout:2 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 10 librados: watch_flush enter 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 10 librados: watch_flush exit 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 10 client.4131.objecter successfully canceled tick 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 10 monclient: shutdown 2024-03-29T10:36:53.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 >> v1:172.21.15.151:6789/0 conn(0x7f17dc09ac40 legacy=0x7f17dc1da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 shutdown_connections 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 >> 172.21.15.169:0/4038784799 conn(0x7f17dc066b00 msgr2=0x7f17dc1cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 shutdown_connections 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.780+0000 7f17e124e640 1 -- 172.21.15.169:0/4038784799 wait complete. 2024-03-29T10:36:53.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.781+0000 7f17e124e640 1 librados: shutdown 2024-03-29T10:36:53.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new b5ed214d-a882-467f-8676-a93556702bd4 3 2024-03-29T10:36:53.790 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.805+0000 7fd297238040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:36:53.802 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.818+0000 7f31d2e45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:36:53.845 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.860+0000 7f31d2e45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:36:53.847 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.862+0000 7fd297238040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:36:53.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.890+0000 7f5de6df6640 10 monclient: get_monmap_and_config 2024-03-29T10:36:53.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.890+0000 7f5de6df6640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.890+0000 7f5de6df6640 10 monclient: monmap: 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:53.891958+0000 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:53.891958+0000 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:53.876 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 1 Processor -- start 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 1 -- start start 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient: init 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-c con 0x7f5de009ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-b con 0x7f5de009c870 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-a con 0x7f5de00b4360 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de009dec0 con 0x7f5de00b4360 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de009da30 con 0x7f5de009ac40 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de00b39c0 con 0x7f5de009c870 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de6df6640 10 monclient(hunting): authenticate will time out at 942.177746s 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de4b92640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5de009ac40 0x7f5de00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50464/0 (socket says 172.21.15.169:50464) 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.891+0000 7f5de4b92640 1 -- 172.21.15.169:0/1156916169 learned_addr learned my addr 172.21.15.169:0/1156916169 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881270043 0 0) 0x7f5de009da30 con 0x7f5de009ac40 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:36:53.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd0003520 con 0x7f5de009ac40 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108751826 0 0) 0x7f5de00b39c0 con 0x7f5de009c870 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de009da30 con 0x7f5de009c870 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574254766 0 0) 0x7f5dd8001490 con 0x7f5de00b4360 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de00b39c0 con 0x7f5de00b4360 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817403942 0 0) 0x7f5dd0003520 con 0x7f5de009ac40 2024-03-29T10:36:53.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 >> v1:172.21.15.151:6790/0 conn(0x7f5de009c870 legacy=0x7f5de013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 >> v1:172.21.15.151:6789/0 conn(0x7f5de00b4360 legacy=0x7f5de014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 monclient: found mon.noname-c 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de00b5160 con 0x7f5de009ac40 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dd4001690 con 0x7f5de009ac40 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5de6df6640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5de6df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: dump: 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5de6df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dd4001870 con 0x7f5de009ac40 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 1 -- 172.21.15.169:0/1156916169 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dd4001c80 con 0x7f5de009ac40 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5de6df6640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5de6df6640 4 set_mon_vals no callback set 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.892+0000 7f5ddf7fe640 10 monclient: dump: 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.890 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 10 monclient: shutdown 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 1 -- 172.21.15.169:0/1156916169 >> v1:172.21.15.169:6789/0 conn(0x7f5de009ac40 legacy=0x7f5de00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 1 -- 172.21.15.169:0/1156916169 shutdown_connections 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 1 -- 172.21.15.169:0/1156916169 >> 172.21.15.169:0/1156916169 conn(0x7f5de0066b00 msgr2=0x7f5de0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 1 -- 172.21.15.169:0/1156916169 shutdown_connections 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.893+0000 7f5de6df6640 1 -- 172.21.15.169:0/1156916169 wait complete. 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 10 monclient: build_initial_monmap 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 10 monclient: monmap: 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 librados: starting msgr at 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 librados: starting objecter 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 Processor -- start 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 -- start start 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 librados: setting wanted keys 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 1 librados: calling monclient init 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.894+0000 7f5de6df6640 10 monclient: init 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:53.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-c con 0x7f5de009ac40 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-b con 0x7f5de01d7f60 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient(hunting): picked mon.noname-a con 0x7f5de01398c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de00b18b0 con 0x7f5de01398c0 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de01d73c0 con 0x7f5de01d7f60 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de00b1ae0 con 0x7f5de009ac40 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5de6df6640 10 monclient(hunting): authenticate will time out at 942.181657s 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5de01d7f60 0x7f5de01391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50476/0 (socket says 172.21.15.169:50476) 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddffff640 1 -- 172.21.15.169:0/3230489610 learned_addr learned my addr 172.21.15.169:0/3230489610 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter ms_handle_connect 0x7f5de01d7f60 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter ms_handle_connect 0x7f5de01398c0 2024-03-29T10:36:53.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter ms_handle_connect 0x7f5de009ac40 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.895+0000 7f5ddd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723535911 0 0) 0x7f5de01d73c0 con 0x7f5de01d7f60 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc4004eb0 con 0x7f5de01d7f60 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000216938 0 0) 0x7f5de00b18b0 con 0x7f5de01398c0 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:36:53.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de01d73c0 con 0x7f5de01398c0 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652768994 0 0) 0x7f5de00b1ae0 con 0x7f5de009ac40 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de00b18b0 con 0x7f5de009ac40 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647831538 0 0) 0x7f5dc4004eb0 con 0x7f5de01d7f60 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5de00b1ae0 con 0x7f5de01d7f60 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dcc0019b0 con 0x7f5de01d7f60 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): dump: 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.895 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801752229 0 0) 0x7f5de01d73c0 con 0x7f5de01398c0 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc4004eb0 con 0x7f5de01398c0 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dd8001b40 con 0x7f5de01398c0 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:53.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): dump: 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72233377 0 0) 0x7f5de00b18b0 con 0x7f5de009ac40 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5de01d73c0 con 0x7f5de009ac40 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dd4001b80 con 0x7f5de009ac40 2024-03-29T10:36:53.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient(hunting): dump: 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 577373547 0 0) 0x7f5de00b1ae0 con 0x7f5de01d7f60 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 >> v1:172.21.15.151:6790/0 conn(0x7f5de009ac40 legacy=0x7f5de01d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 >> v1:172.21.15.151:6789/0 conn(0x7f5de01398c0 legacy=0x7f5de00b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 monclient: found mon.b 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de00acd30 con 0x7f5de01d7f60 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 10 monclient: _renew_subs 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de00b0830 con 0x7f5de01d7f60 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 10 monclient: _renew_subs 2024-03-29T10:36:53.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de01e3bc0 con 0x7f5de01d7f60 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5de6df6640 1 librados: init done 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dcc003610 con 0x7f5de01d7f60 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddeffd640 4 set_mon_vals no callback set 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dcc0039a0 con 0x7f5de01d7f60 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 monclient: dump: 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:53.900 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5dcc003f50 con 0x7f5de01d7f60 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 252543531 0 0) 0x7f5dcc006610 con 0x7f5de01d7f60 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 10 client.4132.objecter ms_dispatch 0x7f5de0000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.896+0000 7f5ddd7fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.897+0000 7f5de6df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.897+0000 7f5de6df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.897+0000 7f5de6df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.897+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0005080 con 0x7f5de01d7f60 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.899+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5dcc006180 con 0x7f5de01d7f60 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.899+0000 7f5ddd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:53.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.899+0000 7f5ddd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:53.902 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:53.908+0000 7fd297238040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:36:53.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.969+0000 7f5de6df6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b5ed214d-a882-467f-8676-a93556702bd4", "id": 3}] 2024-03-29T10:36:53.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.969+0000 7f5de6df6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b5ed214d-a882-467f-8676-a93556702bd4", "id": 3}] 2024-03-29T10:36:53.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.969+0000 7f5de6df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:53.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.969+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b5ed214d-a882-467f-8676-a93556702bd4", "id": 3} v 0) v1 -- 0x7f5dc00058b0 con 0x7f5de01d7f60 2024-03-29T10:36:53.961 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:53.977+0000 7f31d2e45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:36:53.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.977+0000 7f5ddd7fa640 1 -- 172.21.15.169:0/3230489610 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b5ed214d-a882-467f-8676-a93556702bd4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1909239001 0 525046306) 0x7f5dcc0063e0 con 0x7f5de01d7f60 2024-03-29T10:36:53.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.977+0000 7f5ddd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b5ed214d-a882-467f-8676-a93556702bd4", "id": 3}] 2024-03-29T10:36:53.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.977+0000 7f5ddd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:53.962 INFO:teuthology.orchestra.run.smithi169.stdout:3 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 10 librados: watch_flush enter 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 10 librados: watch_flush exit 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 10 client.4132.objecter successfully canceled tick 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 10 monclient: shutdown 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 >> v1:172.21.15.169:6789/0 conn(0x7f5de01d7f60 legacy=0x7f5de01391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 shutdown_connections 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 >> 172.21.15.169:0/3230489610 conn(0x7f5de0066b00 msgr2=0x7f5de01420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 shutdown_connections 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 -- 172.21.15.169:0/3230489610 wait complete. 2024-03-29T10:36:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:53.978+0000 7f5de6df6640 1 librados: shutdown 2024-03-29T10:36:53.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new efc0ecc0-1f5f-484f-921a-7c52e0c0b360 4 2024-03-29T10:36:54.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.032+0000 7fd297238040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:36:54.030 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.046+0000 7f31d2e45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:36:54.076 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.092+0000 7f31d2e45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:36:54.090 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.105+0000 7fd297238040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.133+0000 7fcc801c6640 10 monclient: get_monmap_and_config 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.133+0000 7fcc801c6640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.133+0000 7fcc801c6640 10 monclient: monmap: 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:54.134847+0000 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:54.134847+0000 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:54.119 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.133+0000 7fcc801c6640 1 Processor -- start 2024-03-29T10:36:54.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 1 -- start start 2024-03-29T10:36:54.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient: init 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc7809ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7809c870 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc780b4360 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7809dec0 con 0x7fcc7809c870 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7809da30 con 0x7fcc7809ac40 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780b39c0 con 0x7fcc780b4360 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.130 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc801c6640 10 monclient(hunting): authenticate will time out at 942.420561s 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc7df62640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcc7809ac40 0x7fcc780b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50492/0 (socket says 172.21.15.169:50492) 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.134+0000 7fcc7df62640 1 -- 172.21.15.169:0/2045259054 learned_addr learned my addr 172.21.15.169:0/2045259054 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467474122 0 0) 0x7fcc7809da30 con 0x7fcc7809ac40 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc64003520 con 0x7fcc7809ac40 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790763026 0 0) 0x7fcc68001040 con 0x7fcc7809c870 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc7809da30 con 0x7fcc7809c870 2024-03-29T10:36:54.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731858726 0 0) 0x7fcc780b39c0 con 0x7fcc780b4360 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68001040 con 0x7fcc780b4360 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640110102 0 0) 0x7fcc64003520 con 0x7fcc7809ac40 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 >> v1:172.21.15.151:6790/0 conn(0x7fcc780b4360 legacy=0x7fcc7814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 >> v1:172.21.15.151:6789/0 conn(0x7fcc7809c870 legacy=0x7fcc7813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 monclient: found mon.noname-c 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780b5160 con 0x7fcc7809ac40 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc74001690 con 0x7fcc7809ac40 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc801c6640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc801c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: dump: 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.133 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc801c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc74001870 con 0x7fcc7809ac40 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 1 -- 172.21.15.169:0/2045259054 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc74001c80 con 0x7fcc7809ac40 2024-03-29T10:36:54.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc7cf60640 10 monclient: dump: 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc801c6640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.135+0000 7fcc801c6640 4 set_mon_vals no callback set 2024-03-29T10:36:54.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 10 monclient: shutdown 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 1 -- 172.21.15.169:0/2045259054 >> v1:172.21.15.169:6789/0 conn(0x7fcc7809ac40 legacy=0x7fcc780b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 1 -- 172.21.15.169:0/2045259054 shutdown_connections 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 1 -- 172.21.15.169:0/2045259054 >> 172.21.15.169:0/2045259054 conn(0x7fcc78066b00 msgr2=0x7fcc78066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 1 -- 172.21.15.169:0/2045259054 shutdown_connections 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.136+0000 7fcc801c6640 1 -- 172.21.15.169:0/2045259054 wait complete. 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 10 monclient: monmap: 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.136 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 librados: starting msgr at 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 librados: starting objecter 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 Processor -- start 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 -- start start 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 librados: setting wanted keys 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 1 librados: calling monclient init 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.137+0000 7fcc801c6640 10 monclient: init 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc7809ac40 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc780b1280 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc781d6ed0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc781422a0 con 0x7fcc780b1280 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc78142f50 con 0x7fcc781d6ed0 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc781ccab0 con 0x7fcc7809ac40 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc801c6640 10 monclient(hunting): authenticate will time out at 942.424626s 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc7e763640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcc781d6ed0 0x7fcc7813adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50506/0 (socket says 172.21.15.169:50506) 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc7e763640 1 -- 172.21.15.169:0/4206422727 learned_addr learned my addr 172.21.15.169:0/4206422727 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter ms_handle_connect 0x7fcc781d6ed0 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter ms_handle_connect 0x7fcc780b1280 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter ms_handle_connect 0x7fcc7809ac40 2024-03-29T10:36:54.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.138+0000 7fcc6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138293844 0 0) 0x7fcc78142f50 con 0x7fcc781d6ed0 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5c004eb0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828598443 0 0) 0x7fcc781422a0 con 0x7fcc780b1280 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc78142f50 con 0x7fcc780b1280 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397865127 0 0) 0x7fcc781ccab0 con 0x7fcc7809ac40 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.139 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc781422a0 con 0x7fcc7809ac40 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152977572 0 0) 0x7fcc5c004eb0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc781ccab0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc70001c70 con 0x7fcc781d6ed0 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.140 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842270491 0 0) 0x7fcc78142f50 con 0x7fcc780b1280 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc5c004eb0 con 0x7fcc780b1280 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc68001850 con 0x7fcc780b1280 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.141 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3458595118 0 0) 0x7fcc781422a0 con 0x7fcc7809ac40 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc78142f50 con 0x7fcc7809ac40 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc74001b80 con 0x7fcc7809ac40 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338231908 0 0) 0x7fcc781ccab0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 >> v1:172.21.15.151:6790/0 conn(0x7fcc7809ac40 legacy=0x7fcc78141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 >> v1:172.21.15.151:6789/0 conn(0x7fcc780b1280 legacy=0x7fcc78142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 monclient: found mon.b 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc781ccc40 con 0x7fcc781d6ed0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 10 monclient: _renew_subs 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7813b4d0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 10 monclient: _renew_subs 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc781e3720 con 0x7fcc781d6ed0 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc801c6640 1 librados: init done 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc70003990 con 0x7fcc781d6ed0 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcc70003de0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 monclient: dump: 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.145 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6ffff640 4 set_mon_vals no callback set 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcc70004390 con 0x7fcc781d6ed0 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3546547452 0 0) 0x7fcc70006ae0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 10 client.4138.objecter ms_dispatch 0x7fcc78000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.139+0000 7fcc6e7fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.140+0000 7fcc801c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.140+0000 7fcc801c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.140+0000 7fcc801c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.140+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60005080 con 0x7fcc781d6ed0 2024-03-29T10:36:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.142+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcc700065d0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.142+0000 7fcc6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.147 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.142+0000 7fcc6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:54.147 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.155+0000 7fd297238040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:36:54.147 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.145+0000 7f31d2e45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:36:54.176 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.191+0000 7f31d2e45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:36:54.193 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.208+0000 7fd297238040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:36:54.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.213+0000 7fcc801c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "efc0ecc0-1f5f-484f-921a-7c52e0c0b360", "id": 4}] 2024-03-29T10:36:54.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.213+0000 7fcc801c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "efc0ecc0-1f5f-484f-921a-7c52e0c0b360", "id": 4}] 2024-03-29T10:36:54.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.213+0000 7fcc801c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.213+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "efc0ecc0-1f5f-484f-921a-7c52e0c0b360", "id": 4} v 0) v1 -- 0x7fcc600058b0 con 0x7fcc781d6ed0 2024-03-29T10:36:54.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.218+0000 7fcc6e7fc640 1 -- 172.21.15.169:0/4206422727 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efc0ecc0-1f5f-484f-921a-7c52e0c0b360", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2237831582 0 1697012583) 0x7fcc70006770 con 0x7fcc781d6ed0 2024-03-29T10:36:54.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.218+0000 7fcc6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "efc0ecc0-1f5f-484f-921a-7c52e0c0b360", "id": 4}] 2024-03-29T10:36:54.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.218+0000 7fcc6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:54.204 INFO:teuthology.orchestra.run.smithi169.stdout:4 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 10 librados: watch_flush enter 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 10 librados: watch_flush exit 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 10 client.4138.objecter successfully canceled tick 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 10 monclient: shutdown 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 >> v1:172.21.15.169:6789/0 conn(0x7fcc781d6ed0 legacy=0x7fcc7813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 shutdown_connections 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 >> 172.21.15.169:0/4206422727 conn(0x7fcc78066b00 msgr2=0x7fcc78066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 shutdown_connections 2024-03-29T10:36:54.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 -- 172.21.15.169:0/4206422727 wait complete. 2024-03-29T10:36:54.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.219+0000 7fcc801c6640 1 librados: shutdown 2024-03-29T10:36:54.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new eb0fcacd-f425-4f16-91b5-7bdb850ceddc 5 2024-03-29T10:36:54.244 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.258+0000 7fd297238040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.329+0000 7f519083e640 10 monclient: get_monmap_and_config 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.329+0000 7f519083e640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.329+0000 7f519083e640 10 monclient: monmap: 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:54.331283+0000 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:54.331283+0000 2024-03-29T10:36:54.315 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.330+0000 7f519083e640 1 Processor -- start 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.330+0000 7f519083e640 1 -- start start 2024-03-29T10:36:54.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.330+0000 7f519083e640 10 monclient: init 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient(hunting): picked mon.noname-b con 0x7f518c09ac40 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient(hunting): picked mon.noname-c con 0x7f518c09c870 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient(hunting): picked mon.noname-a con 0x7f518c0b4360 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c09dec0 con 0x7f518c0b4360 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c09da30 con 0x7f518c09c870 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b39c0 con 0x7f518c09ac40 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient(hunting): authenticate will time out at 942.616994s 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f5189d9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f518c09c870 0x7f518c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50512/0 (socket says 172.21.15.169:50512) 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f5189d9b640 1 -- 172.21.15.169:0/968647306 learned_addr learned my addr 172.21.15.169:0/968647306 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883657411 0 0) 0x7f518c09da30 con 0x7f518c09c870 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174003520 con 0x7f518c09c870 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871351186 0 0) 0x7f5184001490 con 0x7f518c0b4360 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c09da30 con 0x7f518c0b4360 2024-03-29T10:36:54.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281575793 0 0) 0x7f518c0b39c0 con 0x7f518c09ac40 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5184001490 con 0x7f518c09ac40 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145563548 0 0) 0x7f5174003520 con 0x7f518c09c870 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 >> v1:172.21.15.151:6790/0 conn(0x7f518c09ac40 legacy=0x7f518c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 >> v1:172.21.15.151:6789/0 conn(0x7f518c0b4360 legacy=0x7f518c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 monclient: found mon.noname-c 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0b5160 con 0x7f518c09c870 2024-03-29T10:36:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5180001690 con 0x7f518c09c870 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: dump: 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.327 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5180001870 con 0x7f518c09c870 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 1 -- 172.21.15.169:0/968647306 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5180001c80 con 0x7f518c09c870 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f519083e640 4 set_mon_vals no callback set 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.331+0000 7f518959a640 10 monclient: dump: 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.328 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 10 monclient: shutdown 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 1 -- 172.21.15.169:0/968647306 >> v1:172.21.15.169:6789/0 conn(0x7f518c09c870 legacy=0x7f518c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 1 -- 172.21.15.169:0/968647306 shutdown_connections 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 1 -- 172.21.15.169:0/968647306 >> 172.21.15.169:0/968647306 conn(0x7f518c066b00 msgr2=0x7f518c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 1 -- 172.21.15.169:0/968647306 shutdown_connections 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.332+0000 7f519083e640 1 -- 172.21.15.169:0/968647306 wait complete. 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 10 monclient: monmap: 2024-03-29T10:36:54.329 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 librados: starting msgr at 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 librados: starting objecter 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 Processor -- start 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 -- start start 2024-03-29T10:36:54.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 librados: setting wanted keys 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 1 librados: calling monclient init 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.333+0000 7f519083e640 10 monclient: init 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient(hunting): picked mon.noname-c con 0x7f518c09ac40 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient(hunting): picked mon.noname-a con 0x7f518c0b6b00 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient(hunting): picked mon.noname-b con 0x7f518c1406b0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c1d7e20 con 0x7f518c0b6b00 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c140b00 con 0x7f518c1406b0 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0ad080 con 0x7f518c09ac40 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient(hunting): authenticate will time out at 942.620079s 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f5189d9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f518c0b6b00 0x7f518c13ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.169:43700/0 (socket says 172.21.15.169:43700) 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f5189d9b640 1 -- 172.21.15.169:0/2025148046 learned_addr learned my addr 172.21.15.169:0/2025148046 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter ms_handle_connect 0x7f518c1406b0 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter ms_handle_connect 0x7f518c0b6b00 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter ms_handle_connect 0x7f518c09ac40 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707770048 0 0) 0x7f518c140b00 con 0x7f518c1406b0 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.332 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f516c004eb0 con 0x7f518c1406b0 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128888844 0 0) 0x7f518c1d7e20 con 0x7f518c0b6b00 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c140b00 con 0x7f518c0b6b00 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524575928 0 0) 0x7f518c0ad080 con 0x7f518c09ac40 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c1d7e20 con 0x7f518c09ac40 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895462361 0 0) 0x7f516c004eb0 con 0x7f518c1406b0 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f518c0ad080 con 0x7f518c1406b0 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5184001b00 con 0x7f518c1406b0 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): dump: 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.334 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365990005 0 0) 0x7f518c140b00 con 0x7f518c0b6b00 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f516c004eb0 con 0x7f518c0b6b00 2024-03-29T10:36:54.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5180001b80 con 0x7f518c0b6b00 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): dump: 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193786433 0 0) 0x7f518c1d7e20 con 0x7f518c09ac40 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f518c140b00 con 0x7f518c09ac40 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f517c001830 con 0x7f518c09ac40 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient(hunting): dump: 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.337 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203203228 0 0) 0x7f518c0ad080 con 0x7f518c1406b0 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 >> v1:172.21.15.151:6790/0 conn(0x7f518c09ac40 legacy=0x7f518c1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 >> v1:172.21.15.151:6789/0 conn(0x7f518c0b6b00 legacy=0x7f518c13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 monclient: found mon.b 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c1d7ab0 con 0x7f518c1406b0 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f517affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:36:54.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient: _renew_subs 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.334+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c1e3570 con 0x7f518c1406b0 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5184001ce0 con 0x7f518c1406b0 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5184003b10 con 0x7f518c1406b0 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f519083e640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 10 monclient: dump: 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.339 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f5188d99640 4 set_mon_vals no callback set 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f519083e640 10 monclient: _renew_subs 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f519083e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c1e38e0 con 0x7f518c1406b0 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f519083e640 1 librados: init done 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5184004080 con 0x7f518c1406b0 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 4232746893 0 0) 0x7f51840048b0 con 0x7f518c1406b0 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 10 client.4144.objecter ms_dispatch 0x7f518c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:36:54.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.335+0000 7f517affd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.336+0000 7f519083e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.336+0000 7f519083e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.336+0000 7f519083e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.336+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c09b020 con 0x7f518c1406b0 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.338+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5184004a70 con 0x7f518c1406b0 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.338+0000 7f517affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.341 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.338+0000 7f517affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.408+0000 7f519083e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb0fcacd-f425-4f16-91b5-7bdb850ceddc", "id": 5}] 2024-03-29T10:36:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.408+0000 7f519083e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb0fcacd-f425-4f16-91b5-7bdb850ceddc", "id": 5}] 2024-03-29T10:36:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.408+0000 7f519083e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.408+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eb0fcacd-f425-4f16-91b5-7bdb850ceddc", "id": 5} v 0) v1 -- 0x7f518c04f730 con 0x7f518c1406b0 2024-03-29T10:36:54.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.416+0000 7f517affd640 1 -- 172.21.15.169:0/2025148046 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb0fcacd-f425-4f16-91b5-7bdb850ceddc", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3853760944 0 1988412176) 0x7f5184004330 con 0x7f518c1406b0 2024-03-29T10:36:54.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.416+0000 7f517affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb0fcacd-f425-4f16-91b5-7bdb850ceddc", "id": 5}] 2024-03-29T10:36:54.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.416+0000 7f517affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:54.402 INFO:teuthology.orchestra.run.smithi169.stdout:5 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 10 librados: watch_flush enter 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 10 librados: watch_flush exit 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 10 client.4144.objecter successfully canceled tick 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 10 monclient: shutdown 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 >> v1:172.21.15.169:6789/0 conn(0x7f518c1406b0 legacy=0x7f518c1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 shutdown_connections 2024-03-29T10:36:54.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 >> 172.21.15.169:0/2025148046 conn(0x7f518c066b00 msgr2=0x7f518c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 shutdown_connections 2024-03-29T10:36:54.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 -- 172.21.15.169:0/2025148046 wait complete. 2024-03-29T10:36:54.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.417+0000 7f519083e640 1 librados: shutdown 2024-03-29T10:36:54.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 6c821dc3-03d5-446b-adc2-a299de39985f 6 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 10 monclient: get_monmap_and_config 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 10 monclient: monmap: 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.554 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:54.570685+0000 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:54.570685+0000 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:54.555 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.561 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.572+0000 7f31d2e45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 1 Processor -- start 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 1 -- start start 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.569+0000 7f1ef2b1c640 10 monclient: init 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:54.564 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec09ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f1eec09c870 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec0b4360 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec09dec0 con 0x7f1eec09c870 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec09da30 con 0x7f1eec09ac40 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0b39c0 con 0x7f1eec0b4360 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef2b1c640 10 monclient(hunting): authenticate will time out at 942.856403s 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef08b8640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1eec09ac40 0x7f1eec0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:57460/0 (socket says 172.21.15.169:57460) 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1ef08b8640 1 -- 172.21.15.169:0/2707329847 learned_addr learned my addr 172.21.15.169:0/2707329847 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948072045 0 0) 0x7f1eec09da30 con 0x7f1eec09ac40 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc003520 con 0x7f1eec09ac40 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.570+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933150119 0 0) 0x7f1ed8001110 con 0x7f1eec09c870 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec09da30 con 0x7f1eec09c870 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297426262 0 0) 0x7f1eec0b39c0 con 0x7f1eec0b4360 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:36:54.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed8001110 con 0x7f1eec0b4360 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867617829 0 0) 0x7f1edc003520 con 0x7f1eec09ac40 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 >> v1:172.21.15.151:6790/0 conn(0x7f1eec0b4360 legacy=0x7f1eec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 >> v1:172.21.15.151:6789/0 conn(0x7f1eec09c870 legacy=0x7f1eec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 monclient: found mon.noname-c 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0b5160 con 0x7f1eec09ac40 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1ee0001690 con 0x7f1eec09ac40 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: dump: 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee0001870 con 0x7f1eec09ac40 2024-03-29T10:36:54.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 1 -- 172.21.15.169:0/2707329847 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1ee0001c80 con 0x7f1eec09ac40 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 4 set_mon_vals no callback set 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1eeb7fe640 10 monclient: dump: 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.569 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 10 monclient: shutdown 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/2707329847 >> v1:172.21.15.169:6789/0 conn(0x7f1eec09ac40 legacy=0x7f1eec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/2707329847 shutdown_connections 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/2707329847 >> 172.21.15.169:0/2707329847 conn(0x7f1eec066b00 msgr2=0x7f1eec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/2707329847 shutdown_connections 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.571+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/2707329847 wait complete. 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 10 monclient: monmap: 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.570 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 librados: starting msgr at 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 librados: starting objecter 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 Processor -- start 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 -- start start 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 librados: setting wanted keys 2024-03-29T10:36:54.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 1 librados: calling monclient init 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.572+0000 7f1ef2b1c640 10 monclient: init 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec09ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec0b1280 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f1eec1d6ed0 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec1422a0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec142f50 con 0x7f1eec09ac40 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec1ccab0 con 0x7f1eec0b1280 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef2b1c640 10 monclient(hunting): authenticate will time out at 942.859444s 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef08b8640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1eec09ac40 0x7f1eec141ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:57474/0 (socket says 172.21.15.169:57474) 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ef08b8640 1 -- 172.21.15.169:0/137114793 learned_addr learned my addr 172.21.15.169:0/137114793 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter ms_handle_connect 0x7f1eec09ac40 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter ms_handle_connect 0x7f1eec1d6ed0 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter ms_handle_connect 0x7f1eec0b1280 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723159994 0 0) 0x7f1eec142f50 con 0x7f1eec09ac40 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed0004eb0 con 0x7f1eec09ac40 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832230256 0 0) 0x7f1eec1422a0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec142f50 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205795678 0 0) 0x7f1eec1ccab0 con 0x7f1eec0b1280 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.573+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec1422a0 con 0x7f1eec0b1280 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1127026998 0 0) 0x7f1eec142f50 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eec1ccab0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1ee4001b50 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): dump: 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 225723245 0 0) 0x7f1ed0004eb0 con 0x7f1eec09ac40 2024-03-29T10:36:54.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eec142f50 con 0x7f1eec09ac40 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1ee0001bc0 con 0x7f1eec09ac40 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient(hunting): dump: 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.576 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3551847763 0 0) 0x7f1eec1ccab0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 >> v1:172.21.15.151:6790/0 conn(0x7f1eec0b1280 legacy=0x7f1eec142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 >> v1:172.21.15.169:6789/0 conn(0x7f1eec09ac40 legacy=0x7f1eec141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 monclient: found mon.a 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec1ccc40 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 5 monclient: authenticate success, global_id 4155 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 10 monclient: _renew_subs 2024-03-29T10:36:54.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec13b4d0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 10 monclient: _renew_subs 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec1e3720 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ef2b1c640 1 librados: init done 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee4002060 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1ee4003b30 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 monclient: dump: 2024-03-29T10:36:54.578 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1eeaffd640 4 set_mon_vals no callback set 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1ee4004020 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 764533077 0 0) 0x7f1ee40049c0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 10 client.4155.objecter ms_dispatch 0x7f1eec000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:36:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.574+0000 7f1ee97fa640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.575+0000 7f1ef2b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.575+0000 7f1ef2b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.575+0000 7f1ef2b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.575+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec1cce30 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.577+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1ee40042d0 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.577+0000 7f1ee97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.577+0000 7f1ee97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:54.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.649+0000 7f1ef2b1c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6c821dc3-03d5-446b-adc2-a299de39985f", "id": 6}] 2024-03-29T10:36:54.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.649+0000 7f1ef2b1c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6c821dc3-03d5-446b-adc2-a299de39985f", "id": 6}] 2024-03-29T10:36:54.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.649+0000 7f1ef2b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:54.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.649+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6c821dc3-03d5-446b-adc2-a299de39985f", "id": 6} v 0) v1 -- 0x7f1eec09b020 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ee97fa640 1 -- 172.21.15.169:0/137114793 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c821dc3-03d5-446b-adc2-a299de39985f", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3868361264 0 1113811849) 0x7f1ee4004570 con 0x7f1eec1d6ed0 2024-03-29T10:36:54.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ee97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6c821dc3-03d5-446b-adc2-a299de39985f", "id": 6}] 2024-03-29T10:36:54.636 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ee97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:54.636 INFO:teuthology.orchestra.run.smithi169.stdout:6 2024-03-29T10:36:54.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ef2b1c640 10 librados: watch_flush enter 2024-03-29T10:36:54.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ef2b1c640 10 librados: watch_flush exit 2024-03-29T10:36:54.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ef2b1c640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-29T10:36:54.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.651+0000 7f1ef2b1c640 10 client.4155.objecter successfully canceled tick 2024-03-29T10:36:54.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 10 monclient: shutdown 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 >> v1:172.21.15.151:6789/0 conn(0x7f1eec1d6ed0 legacy=0x7f1eec13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 shutdown_connections 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 >> 172.21.15.169:0/137114793 conn(0x7f1eec066b00 msgr2=0x7f1eec066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 shutdown_connections 2024-03-29T10:36:54.638 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 -- 172.21.15.169:0/137114793 wait complete. 2024-03-29T10:36:54.639 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.652+0000 7f1ef2b1c640 1 librados: shutdown 2024-03-29T10:36:54.641 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.656+0000 7fd297238040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:36:54.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new a602a4f1-35fd-4ea9-b720-b4d97b5abaaa 7 2024-03-29T10:36:54.694 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.710+0000 7f31d2e45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 10 monclient: get_monmap_and_config 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 10 monclient: monmap: 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:54.763664+0000 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:54.763664+0000 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.747 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.748 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.748 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:54.748 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:54.748 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 1 Processor -- start 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 1 -- start start 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.762+0000 7f66d59ed640 10 monclient: init 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-c con 0x7f66d009ac40 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-b con 0x7f66d009c870 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-a con 0x7f66d00b4360 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d009dec0 con 0x7f66d00b4360 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d009da30 con 0x7f66d009ac40 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00b39c0 con 0x7f66d009c870 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 10 monclient(hunting): authenticate will time out at 943.049227s 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66ceffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f66d009ac40 0x7f66d00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:57486/0 (socket says 172.21.15.169:57486) 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66ceffd640 1 -- 172.21.15.169:0/1323341550 learned_addr learned my addr 172.21.15.169:0/1323341550 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36778221 0 0) 0x7f66d009da30 con 0x7f66d009ac40 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b8003520 con 0x7f66d009ac40 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650895993 0 0) 0x7f66d00b39c0 con 0x7f66d009c870 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d009da30 con 0x7f66d009c870 2024-03-29T10:36:54.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981756772 0 0) 0x7f66c8001490 con 0x7f66d00b4360 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00b39c0 con 0x7f66d00b4360 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217867633 0 0) 0x7f66b8003520 con 0x7f66d009ac40 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 >> v1:172.21.15.151:6790/0 conn(0x7f66d009c870 legacy=0x7f66d013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 >> v1:172.21.15.151:6789/0 conn(0x7f66d00b4360 legacy=0x7f66d014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 monclient: found mon.noname-c 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:36:54.759 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00b5160 con 0x7f66d009ac40 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c4001690 con 0x7f66d009ac40 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66cdffb640 10 monclient: dump: 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.760 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.763+0000 7f66d59ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66c4001870 con 0x7f66d009ac40 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 1 -- 172.21.15.169:0/1323341550 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c4001c80 con 0x7f66d009ac40 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 4 set_mon_vals no callback set 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66cdffb640 10 monclient: dump: 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.761 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 10 monclient: shutdown 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 1 -- 172.21.15.169:0/1323341550 >> v1:172.21.15.169:6789/0 conn(0x7f66d009ac40 legacy=0x7f66d00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 1 -- 172.21.15.169:0/1323341550 shutdown_connections 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 1 -- 172.21.15.169:0/1323341550 >> 172.21.15.169:0/1323341550 conn(0x7f66d0066b00 msgr2=0x7f66d0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 1 -- 172.21.15.169:0/1323341550 shutdown_connections 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.764+0000 7f66d59ed640 1 -- 172.21.15.169:0/1323341550 wait complete. 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 10 monclient: monmap: 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 librados: starting msgr at 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 librados: starting objecter 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 Processor -- start 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 -- start start 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 librados: setting wanted keys 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 1 librados: calling monclient init 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.765+0000 7f66d59ed640 10 monclient: init 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-a con 0x7f66d009ac40 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-c con 0x7f66d01d7220 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient(hunting): picked mon.noname-b con 0x7f66d01dfb50 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d0142eb0 con 0x7f66d009ac40 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00b7700 con 0x7f66d01dfb50 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00b7970 con 0x7f66d01d7220 2024-03-29T10:36:54.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66d59ed640 10 monclient(hunting): authenticate will time out at 943.052179s 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66cf7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f66d01dfb50 0x7f66d01e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:57488/0 (socket says 172.21.15.169:57488) 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66cf7fe640 1 -- 172.21.15.169:0/1611931361 learned_addr learned my addr 172.21.15.169:0/1611931361 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter ms_handle_connect 0x7f66d01dfb50 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter ms_handle_connect 0x7f66d009ac40 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter ms_handle_connect 0x7f66d01d7220 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83047407 0 0) 0x7f66d00b7970 con 0x7f66d01d7220 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66bc004eb0 con 0x7f66d01d7220 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411813302 0 0) 0x7f66d00b7700 con 0x7f66d01dfb50 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00b7970 con 0x7f66d01dfb50 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689836717 0 0) 0x7f66d0142eb0 con 0x7f66d009ac40 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00b7700 con 0x7f66d009ac40 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 276527519 0 0) 0x7f66bc004eb0 con 0x7f66d01d7220 2024-03-29T10:36:54.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d0142eb0 con 0x7f66d01d7220 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c00018b0 con 0x7f66d01d7220 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): dump: 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611438435 0 0) 0x7f66d00b7700 con 0x7f66d009ac40 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66bc004eb0 con 0x7f66d009ac40 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c40019e0 con 0x7f66d009ac40 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.766+0000 7f66b7fff640 10 monclient(hunting): dump: 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.768 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699963137 0 0) 0x7f66d00b7970 con 0x7f66d01dfb50 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d00b7700 con 0x7f66d01dfb50 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c8001b00 con 0x7f66d01dfb50 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient(hunting): dump: 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.769 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1097771588 0 0) 0x7f66d0142eb0 con 0x7f66d01d7220 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 >> v1:172.21.15.169:6789/0 conn(0x7f66d01dfb50 legacy=0x7f66d01e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 >> v1:172.21.15.151:6789/0 conn(0x7f66d009ac40 legacy=0x7f66d00b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 monclient: found mon.c 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00b8810 con 0x7f66d01d7220 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 5 monclient: authenticate success, global_id 4154 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 10 monclient: _renew_subs 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d00b1a30 con 0x7f66d01d7220 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 10 monclient: _renew_subs 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d01d7f30 con 0x7f66d01d7220 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66d59ed640 1 librados: init done 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66c00035f0 con 0x7f66d01d7220 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f66c0003760 con 0x7f66d01d7220 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66cd7fa640 4 set_mon_vals no callback set 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 monclient: dump: 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f66c0003c50 con 0x7f66d01d7220 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4103896583 0 0) 0x7f66c0005500 con 0x7f66d01d7220 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 10 client.4154.objecter ms_dispatch 0x7f66d0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:36:54.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 3 client.4154.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 3 client.4154.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.767+0000 7f66b7fff640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.768+0000 7f66d59ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.768+0000 7f66d59ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.768+0000 7f66d59ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.768+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b0005080 con 0x7f66d01d7220 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.770+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f66c0004ea0 con 0x7f66d01d7220 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.770+0000 7f66b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:54.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.770+0000 7f66b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:54.783 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.798+0000 7fd297238040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:36:54.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.840+0000 7f66d59ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a602a4f1-35fd-4ea9-b720-b4d97b5abaaa", "id": 7}] 2024-03-29T10:36:54.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.840+0000 7f66d59ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a602a4f1-35fd-4ea9-b720-b4d97b5abaaa", "id": 7}] 2024-03-29T10:36:54.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.840+0000 7f66d59ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.840+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd new", "uuid": "a602a4f1-35fd-4ea9-b720-b4d97b5abaaa", "id": 7} v 0) v1 -- 0x7f66b00058b0 con 0x7f66d01d7220 2024-03-29T10:36:54.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.843+0000 7f66b7fff640 1 -- 172.21.15.169:0/1611931361 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a602a4f1-35fd-4ea9-b720-b4d97b5abaaa", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3910991697 0 1371665406) 0x7f66c0005110 con 0x7f66d01d7220 2024-03-29T10:36:54.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.843+0000 7f66b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a602a4f1-35fd-4ea9-b720-b4d97b5abaaa", "id": 7}] 2024-03-29T10:36:54.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.843+0000 7f66b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:54.828 INFO:teuthology.orchestra.run.smithi169.stdout:7 2024-03-29T10:36:54.829 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.844+0000 7f31d2e45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 10 librados: watch_flush enter 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 10 librados: watch_flush exit 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 10 client.4154.objecter successfully canceled tick 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 10 monclient: shutdown 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 >> v1:172.21.15.151:6790/0 conn(0x7f66d01d7220 legacy=0x7f66d01d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 shutdown_connections 2024-03-29T10:36:54.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 >> 172.21.15.169:0/1611931361 conn(0x7f66d0066b00 msgr2=0x7f66d013c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 shutdown_connections 2024-03-29T10:36:54.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 -- 172.21.15.169:0/1611931361 wait complete. 2024-03-29T10:36:54.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-29T10:36:54.844+0000 7f66d59ed640 1 librados: shutdown 2024-03-29T10:36:54.841 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:36:54.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:36:54.843 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:36:54.843 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:36:54.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:36:54.845 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:36:54.845 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:36:54.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:36:54.847 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:36:54.848 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:36:54.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:36:54.850 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:36:54.850 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:36:54.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:36:54.852 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:36:54.852 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:36:54.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:36:54.854 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:36:54.854 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:36:54.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:36:54.856 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:36:54.856 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:36:54.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:36:54.858 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:36:54.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:54.887 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:54.902+0000 7f31d2e45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:36:54.917 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.932+0000 7fd297238040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:36:54.969 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:54.984+0000 7fd297238040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 10 monclient: get_monmap_and_config 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 10 monclient: monmap: 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.980 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:54.997044+0000 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:54.997044+0000 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:54.981 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 1 Processor -- start 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 1 -- start start 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.996+0000 7fad5ffc4640 10 monclient: init 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-b con 0x7fad580b8240 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-c con 0x7fad580b1f60 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-a con 0x7fad5809c8c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5809bd80 con 0x7fad5809c8c0 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5809b960 con 0x7fad580b1f60 2024-03-29T10:36:54.990 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580b2ec0 con 0x7fad580b8240 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5ffc4640 10 monclient(hunting): authenticate will time out at 946.167014s 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5e561640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fad5809c8c0 0x7fad5814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45814/0 (socket says 172.21.15.151:45814) 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5e561640 1 -- 172.21.15.151:0/753136346 learned_addr learned my addr 172.21.15.151:0/753136346 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5dd60640 1 --1- 172.21.15.151:0/753136346 >> v1:172.21.15.151:6790/0 conn(0x7fad580b8240 0x7fad5813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60984/0 (socket says 172.21.15.151:60984) 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530180086 0 0) 0x7fad580b2ec0 con 0x7fad580b8240 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44003520 con 0x7fad580b8240 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772878032 0 0) 0x7fad5809bd80 con 0x7fad5809c8c0 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.991 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.997+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580b2ec0 con 0x7fad5809c8c0 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202289887 0 0) 0x7fad5809b960 con 0x7fad580b1f60 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5809bd80 con 0x7fad580b1f60 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2627326473 0 0) 0x7fad44003520 con 0x7fad580b8240 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 >> v1:172.21.15.169:6789/0 conn(0x7fad580b1f60 legacy=0x7fad5809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 >> v1:172.21.15.151:6789/0 conn(0x7fad5809c8c0 legacy=0x7fad5814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: _finish_hunting 0 2024-03-29T10:36:54.992 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 monclient: found mon.noname-b 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5813fb60 con 0x7fad580b8240 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: _finish_auth 0 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: _check_auth_tickets 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad48001690 con 0x7fad580b8240 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 5 monclient: authenticate success, global_id 4181 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: dump: 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad48001870 con 0x7fad580b8240 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 1 -- 172.21.15.151:0/753136346 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad48001cc0 con 0x7fad580b8240 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5cd5e640 10 monclient: dump: 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 4 set_mon_vals no callback set 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 10 monclient: shutdown 2024-03-29T10:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 1 -- 172.21.15.151:0/753136346 >> v1:172.21.15.151:6790/0 conn(0x7fad580b8240 legacy=0x7fad5813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.998+0000 7fad5ffc4640 1 -- 172.21.15.151:0/753136346 shutdown_connections 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 -- 172.21.15.151:0/753136346 >> 172.21.15.151:0/753136346 conn(0x7fad58066b00 msgr2=0x7fad58066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 -- 172.21.15.151:0/753136346 shutdown_connections 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 -- 172.21.15.151:0/753136346 wait complete. 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 10 monclient: build_initial_monmap 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 10 monclient: monmap: 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 0 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 0.000000 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:created 0.000000 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:54.996 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 librados: starting msgr at 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 librados: starting objecter 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:54.999+0000 7fad5ffc4640 1 Processor -- start 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 -- start start 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 librados: setting wanted keys 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 librados: calling monclient init 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient: init 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:54.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-b con 0x7fad5809c8c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-a con 0x7fad580b8240 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient(hunting): picked mon.noname-c con 0x7fad5813b1b0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5d55f640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fad580b8240 0x7fad5813a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45820/0 (socket says 172.21.15.151:45820) 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5d55f640 1 -- 172.21.15.151:0/1188097124 learned_addr learned my addr 172.21.15.151:0/1188097124 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad581463a0 con 0x7fad580b8240 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad581db440 con 0x7fad5809c8c0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad581db0d0 con 0x7fad5813b1b0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad5ffc4640 10 monclient(hunting): authenticate will time out at 946.170258s 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter ms_handle_connect 0x7fad580b8240 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter ms_handle_connect 0x7fad5813b1b0 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter ms_handle_connect 0x7fad5809c8c0 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861635296 0 0) 0x7fad581463a0 con 0x7fad580b8240 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3c004eb0 con 0x7fad580b8240 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668997672 0 0) 0x7fad581db0d0 con 0x7fad5813b1b0 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad581463a0 con 0x7fad5813b1b0 2024-03-29T10:36:54.999 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396484252 0 0) 0x7fad581db440 con 0x7fad5809c8c0 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.000+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad581db0d0 con 0x7fad5809c8c0 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573113939 0 0) 0x7fad3c004eb0 con 0x7fad580b8240 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad581db440 con 0x7fad580b8240 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad540019d0 con 0x7fad580b8240 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 243718821 0 0) 0x7fad581db0d0 con 0x7fad5809c8c0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad3c004eb0 con 0x7fad5809c8c0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3439375152 0 0) 0x7fad581463a0 con 0x7fad5813b1b0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad581db0d0 con 0x7fad5813b1b0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad48001110 con 0x7fad5809c8c0 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.001 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad50001a10 con 0x7fad5813b1b0 2024-03-29T10:36:55.002 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662584229 0 0) 0x7fad3c004eb0 con 0x7fad5809c8c0 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 >> v1:172.21.15.151:6790/0 conn(0x7fad5813b1b0 legacy=0x7fad581d3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 >> v1:172.21.15.151:6789/0 conn(0x7fad580b8240 legacy=0x7fad5813a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 monclient: found mon.b 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5813ab20 con 0x7fad5809c8c0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 5 monclient: authenticate success, global_id 4186 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 10 monclient: _renew_subs 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5813add0 con 0x7fad5809c8c0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad48001d00 con 0x7fad5809c8c0 2024-03-29T10:36:55.004 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fad48004cd0 con 0x7fad5809c8c0 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 monclient: dump: 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:epoch 1 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.005 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4ffff640 4 set_mon_vals no callback set 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 10 monclient: _renew_subs 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad581d4870 con 0x7fad5809c8c0 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad5ffc4640 1 librados: init done 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fad480051c0 con 0x7fad5809c8c0 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fad48005b40 con 0x7fad5809c8c0 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 10 client.4186.objecter ms_dispatch 0x7fad58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.001+0000 7fad4e7fc640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.002+0000 7fad5ffc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.002+0000 7fad5ffc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.002+0000 7fad5ffc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.002+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5809b170 con 0x7fad5809c8c0 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.004+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fad48005dc0 con 0x7fad5809c8c0 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.004+0000 7fad4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.007 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.004+0000 7fad4e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:55.026 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:55.041+0000 7f31d2e45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:36:55.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.064+0000 7fad5ffc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:55.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.064+0000 7fad5ffc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:55.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.064+0000 7fad5ffc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:36:55.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.064+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad58142870 con 0x7fad5809c8c0 2024-03-29T10:36:55.049 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.065+0000 7fad4e7fc640 1 -- 172.21.15.151:0/1188097124 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fad58142870 con 0x7fad5809c8c0 2024-03-29T10:36:55.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.065+0000 7fad4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:55.050 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.065+0000 7fad4e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-29T10:36:55.050 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:55.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 10 librados: watch_flush enter 2024-03-29T10:36:55.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 10 librados: watch_flush exit 2024-03-29T10:36:55.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-29T10:36:55.051 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 10 client.4186.objecter successfully canceled tick 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 10 monclient: shutdown 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 >> v1:172.21.15.169:6789/0 conn(0x7fad5809c8c0 legacy=0x7fad581da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 shutdown_connections 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 >> 172.21.15.151:0/1188097124 conn(0x7fad58066b00 msgr2=0x7fad5806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 shutdown_connections 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 -- 172.21.15.151:0/1188097124 wait complete. 2024-03-29T10:36:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-29T10:36:55.066+0000 7fad5ffc4640 1 librados: shutdown 2024-03-29T10:36:55.061 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:36:55.061 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:36:55.081 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:55.096+0000 7fd297238040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:36:55.123 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:55.139+0000 7f31d2e45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:36:55.175 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:55.190+0000 7fd297238040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:36:55.278 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:55.294+0000 7f31d2e45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:36:55.320 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:55.335+0000 7f31d2e45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:36:55.330 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:55.345+0000 7fd297238040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:36:55.372 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:55.386+0000 7fd297238040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:36:55.456 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-29T10:36:55.471+0000 7f9ebc0c8640 -1 Falling back to public interface 2024-03-29T10:36:55.459 INFO:tasks.ceph.osd.1.smithi151.stderr:2024-03-29T10:36:55.474+0000 7f15f0015640 -1 Falling back to public interface 2024-03-29T10:36:55.459 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-29T10:36:55.474+0000 7feb28b36640 -1 Falling back to public interface 2024-03-29T10:36:55.460 INFO:tasks.ceph.osd.0.smithi151.stderr:2024-03-29T10:36:55.476+0000 7fdafda0e640 -1 Falling back to public interface 2024-03-29T10:36:55.463 INFO:tasks.ceph.osd.3.smithi151.stderr:2024-03-29T10:36:55.479+0000 7f118b00d640 -1 Falling back to public interface 2024-03-29T10:36:55.468 INFO:tasks.ceph.osd.2.smithi151.stderr:2024-03-29T10:36:55.484+0000 7f4f0f59a640 -1 Falling back to public interface 2024-03-29T10:36:55.469 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-29T10:36:55.484+0000 7f872b2a2640 -1 Falling back to public interface 2024-03-29T10:36:55.475 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:55.490+0000 7f31d2e45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:36:55.480 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-29T10:36:55.495+0000 7fe8a236b640 -1 Falling back to public interface 2024-03-29T10:36:55.529 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:55.544+0000 7fd297238040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:36:55.867 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:55.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 10 monclient: get_monmap_and_config 2024-03-29T10:36:55.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 10 monclient: build_initial_monmap 2024-03-29T10:36:55.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 10 monclient: monmap: 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:55.974060+0000 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:55.974060+0000 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:36:55.958 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.966 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 1 Processor -- start 2024-03-29T10:36:55.966 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 1 -- start start 2024-03-29T10:36:55.966 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.973+0000 7f1d4ace6640 10 monclient: init 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-b con 0x7f1d441457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-a con 0x7f1d4413f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-c con 0x7f1d440a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d440aaef0 con 0x7f1d4413f650 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d44145bd0 con 0x7f1d440a9d70 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d44145eb0 con 0x7f1d441457f0 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d4ace6640 10 monclient(hunting): authenticate will time out at 947.144017s 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d43fff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1d4413f650 0x7f1d440b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45830/0 (socket says 172.21.15.151:45830) 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d43fff640 1 -- 172.21.15.151:0/3493792050 learned_addr learned my addr 172.21.15.151:0/3493792050 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d48a82640 1 --1- 172.21.15.151:0/3493792050 >> v1:172.21.15.151:6790/0 conn(0x7f1d441457f0 0x7f1d4413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60992/0 (socket says 172.21.15.151:60992) 2024-03-29T10:36:55.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252645012 0 0) 0x7f1d440aaef0 con 0x7f1d4413f650 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d30003520 con 0x7f1d4413f650 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463760243 0 0) 0x7f1d44145eb0 con 0x7f1d441457f0 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d440aaef0 con 0x7f1d441457f0 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.974+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139723840 0 0) 0x7f1d44145bd0 con 0x7f1d440a9d70 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44145eb0 con 0x7f1d440a9d70 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386049876 0 0) 0x7f1d30003520 con 0x7f1d4413f650 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 >> v1:172.21.15.151:6790/0 conn(0x7f1d441457f0 legacy=0x7f1d4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 >> v1:172.21.15.169:6789/0 conn(0x7f1d440a9d70 legacy=0x7f1d4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 monclient: found mon.noname-a 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:55.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44146340 con 0x7f1d4413f650 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d2c001690 con 0x7f1d4413f650 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 5 monclient: authenticate success, global_id 4194 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: dump: 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.970 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d2c001870 con 0x7f1d4413f650 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 1 -- 172.21.15.151:0/3493792050 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d2c001cc0 con 0x7f1d4413f650 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d437fe640 10 monclient: dump: 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.971 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 4 set_mon_vals no callback set 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 10 monclient: shutdown 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 1 -- 172.21.15.151:0/3493792050 >> v1:172.21.15.151:6789/0 conn(0x7f1d4413f650 legacy=0x7f1d440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 1 -- 172.21.15.151:0/3493792050 shutdown_connections 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 1 -- 172.21.15.151:0/3493792050 >> 172.21.15.151:0/3493792050 conn(0x7f1d44066a00 msgr2=0x7f1d44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 1 -- 172.21.15.151:0/3493792050 shutdown_connections 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.975+0000 7f1d4ace6640 1 -- 172.21.15.151:0/3493792050 wait complete. 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 10 monclient: build_initial_monmap 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 10 monclient: monmap: 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 librados: starting msgr at 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 librados: starting objecter 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 Processor -- start 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 -- start start 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 librados: setting wanted keys 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 1 librados: calling monclient init 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.976+0000 7f1d4ace6640 10 monclient: init 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-b con 0x7f1d440a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-a con 0x7f1d4413f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient(hunting): picked mon.noname-c con 0x7f1d441457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d440ac1b0 con 0x7f1d4413f650 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d440ac420 con 0x7f1d440a9d70 2024-03-29T10:36:55.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d44144950 con 0x7f1d441457f0 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d4ace6640 10 monclient(hunting): authenticate will time out at 947.147062s 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d49283640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f1d441457f0 0x7f1d441e1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:32768/0 (socket says 172.21.15.151:32768) 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d49283640 1 -- 172.21.15.151:0/2338746501 learned_addr learned my addr 172.21.15.151:0/2338746501 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter ms_handle_connect 0x7f1d4413f650 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter ms_handle_connect 0x7f1d441457f0 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter ms_handle_connect 0x7f1d440a9d70 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543299320 0 0) 0x7f1d440ac1b0 con 0x7f1d4413f650 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d24004eb0 con 0x7f1d4413f650 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290532661 0 0) 0x7f1d44144950 con 0x7f1d441457f0 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d440ac1b0 con 0x7f1d441457f0 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037944638 0 0) 0x7f1d440ac420 con 0x7f1d440a9d70 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.977+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44144950 con 0x7f1d440a9d70 2024-03-29T10:36:55.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4127295531 0 0) 0x7f1d24004eb0 con 0x7f1d4413f650 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d440ac420 con 0x7f1d4413f650 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d2c001b80 con 0x7f1d4413f650 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): dump: 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.977 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702049488 0 0) 0x7f1d440ac1b0 con 0x7f1d441457f0 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d24004eb0 con 0x7f1d441457f0 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d340018f0 con 0x7f1d441457f0 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): dump: 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.978 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883248824 0 0) 0x7f1d44144950 con 0x7f1d440a9d70 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d440ac1b0 con 0x7f1d440a9d70 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d3c001ae0 con 0x7f1d440a9d70 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient(hunting): dump: 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.979 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1448060282 0 0) 0x7f1d440ac420 con 0x7f1d4413f650 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 >> v1:172.21.15.151:6790/0 conn(0x7f1d441457f0 legacy=0x7f1d441e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 >> v1:172.21.15.169:6789/0 conn(0x7f1d440a9d70 legacy=0x7f1d44141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 monclient: found mon.a 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44144ae0 con 0x7f1d4413f650 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:55.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 5 monclient: authenticate success, global_id 4197 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 10 monclient: _renew_subs 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d441d74e0 con 0x7f1d4413f650 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 10 monclient: _renew_subs 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441d7a10 con 0x7f1d4413f650 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d4ace6640 1 librados: init done 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d2c0010c0 con 0x7f1d4413f650 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d2c004d90 con 0x7f1d4413f650 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:55.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d42ffd640 4 set_mon_vals no callback set 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 monclient: dump: 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1d2c005280 con 0x7f1d4413f650 2024-03-29T10:36:55.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f1d2c006b80 con 0x7f1d4413f650 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 10 client.4197.objecter ms_dispatch 0x7f1d44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.978+0000 7f1d417fa640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.979+0000 7f1d4ace6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.979+0000 7f1d4ace6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.979+0000 7f1d4ace6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.979+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d28005080 con 0x7f1d4413f650 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.981+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1d2c0064b0 con 0x7f1d4413f650 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.981+0000 7f1d417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:55.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:55.981+0000 7f1d417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:56.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d4ace6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d4ace6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d4ace6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d28005330 con 0x7f1d4413f650 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d417fa640 1 -- 172.21.15.151:0/2338746501 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f1d2c006700 con 0x7f1d4413f650 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.043+0000 7f1d417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:36:56.028 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:56.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 10 librados: watch_flush enter 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 10 librados: watch_flush exit 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 10 client.4197.objecter successfully canceled tick 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 10 monclient: shutdown 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 >> v1:172.21.15.151:6789/0 conn(0x7f1d4413f650 legacy=0x7f1d44142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 shutdown_connections 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 >> 172.21.15.151:0/2338746501 conn(0x7f1d44066a00 msgr2=0x7f1d440b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 shutdown_connections 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 -- 172.21.15.151:0/2338746501 wait complete. 2024-03-29T10:36:56.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:36:56.044+0000 7f1d4ace6640 1 librados: shutdown 2024-03-29T10:36:56.039 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:56.294 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:56.310+0000 7f31d2e45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:36:56.348 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:56.363+0000 7fd297238040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:36:56.564 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:56.579+0000 7f31d2e45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:36:56.616 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:56.631+0000 7f31d2e45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:36:56.621 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:56.636+0000 7fd297238040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:36:56.672 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:56.687+0000 7fd297238040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:36:56.729 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:56.744+0000 7f31d2e45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:36:56.789 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-29T10:36:56.804+0000 7fd297238040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:36:57.741 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:57.757+0000 7f3173af4640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:36:57.742 INFO:tasks.ceph.mgr.y.smithi151.stderr:2024-03-29T10:36:57.757+0000 7f3173af4640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:36:58.915 INFO:tasks.ceph.osd.0.smithi151.stderr:2024-03-29T10:36:58.930+0000 7fdafda0e640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:36:58.938 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-29T10:36:58.953+0000 7f9ebc0c8640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:36:58.942 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-29T10:36:58.957+0000 7feb28b36640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:36:58.972 INFO:tasks.ceph.osd.1.smithi151.stderr:2024-03-29T10:36:58.988+0000 7f15f0015640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:36:58.982 INFO:tasks.ceph.osd.2.smithi151.stderr:2024-03-29T10:36:58.998+0000 7f4f0f59a640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:36:58.987 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-29T10:36:59.002+0000 7f872b2a2640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:36:58.992 INFO:tasks.ceph.osd.3.smithi151.stderr:2024-03-29T10:36:59.008+0000 7f118b00d640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:36:59.000 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-29T10:36:59.015+0000 7fe8a236b640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:37:02.845 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:02.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.007+0000 7f7838be8640 10 monclient: get_monmap_and_config 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.007+0000 7f7838be8640 10 monclient: build_initial_monmap 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.007+0000 7f7838be8640 10 monclient: monmap: 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:03.008327+0000 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:03.008327+0000 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:02.993 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:02.994 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:02.994 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:02.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.008+0000 7f7838be8640 1 Processor -- start 2024-03-29T10:37:02.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.008+0000 7f7838be8640 1 -- start start 2024-03-29T10:37:02.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.008+0000 7f7838be8640 10 monclient: init 2024-03-29T10:37:02.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-b con 0x7f783406d9d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-a con 0x7f783413ef30 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-c con 0x7f78340b0860 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78340aed10 con 0x7f783413ef30 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78340ab1b0 con 0x7f78340b0860 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78340ab490 con 0x7f783406d9d0 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.009+0000 7f7838be8640 10 monclient(hunting): authenticate will time out at 954.179211s 2024-03-29T10:37:02.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.010+0000 7f7832d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f78340b0860 0x7f783414ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:54398/0 (socket says 172.21.15.151:54398) 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.010+0000 7f7832d9d640 1 -- 172.21.15.151:0/2390243246 learned_addr learned my addr 172.21.15.151:0/2390243246 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701192847 0 0) 0x7f78340aed10 con 0x7f783413ef30 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c003520 con 0x7f783413ef30 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051636449 0 0) 0x7f78340ab1b0 con 0x7f78340b0860 2024-03-29T10:37:03.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78340aed10 con 0x7f78340b0860 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378535268 0 0) 0x7f78340ab490 con 0x7f783406d9d0 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 10 monclient(hunting): my global_id is 4223 2024-03-29T10:37:03.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.011+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78340ab1b0 con 0x7f783406d9d0 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320186128 0 0) 0x7f78340aed10 con 0x7f78340b0860 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 >> v1:172.21.15.151:6790/0 conn(0x7f783406d9d0 legacy=0x7f783413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 >> v1:172.21.15.151:6789/0 conn(0x7f783413ef30 legacy=0x7f78340af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: _finish_hunting 0 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 monclient: found mon.noname-c 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78340ab770 con 0x7f78340b0860 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: _finish_auth 0 2024-03-29T10:37:03.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: _check_auth_tickets 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f7838be8640 5 monclient: authenticate success, global_id 4231 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f7838be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7824001670 con 0x7f78340b0860 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: dump: 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:03.003 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f7838be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7824001890 con 0x7f78340b0860 2024-03-29T10:37:03.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f783159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f7838be8640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.012+0000 7f7838be8640 4 set_mon_vals no callback set 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f783159a640 1 -- 172.21.15.151:0/2390243246 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7824001c20 con 0x7f78340b0860 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f783159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f783159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f783159a640 10 monclient: dump: 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.005 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f7838be8640 10 monclient: shutdown 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f7838be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:03.006 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f7838be8640 1 -- 172.21.15.151:0/2390243246 >> v1:172.21.15.169:6789/0 conn(0x7f78340b0860 legacy=0x7f783414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.007 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f7838be8640 1 -- 172.21.15.151:0/2390243246 shutdown_connections 2024-03-29T10:37:03.007 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.013+0000 7f7838be8640 1 -- 172.21.15.151:0/2390243246 >> 172.21.15.151:0/2390243246 conn(0x7f783406b660 msgr2=0x7f783406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:03.007 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.014+0000 7f7838be8640 1 -- 172.21.15.151:0/2390243246 shutdown_connections 2024-03-29T10:37:03.007 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.014+0000 7f7838be8640 1 -- 172.21.15.151:0/2390243246 wait complete. 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 10 monclient: build_initial_monmap 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 10 monclient: monmap: 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:03.022 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 librados: starting msgr at 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 librados: starting objecter 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 Processor -- start 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 -- start start 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 librados: setting wanted keys 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 1 librados: calling monclient init 2024-03-29T10:37:03.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.015+0000 7f7838be8640 10 monclient: init 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.016+0000 7f7838be8640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.016+0000 7f7838be8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.016+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-a con 0x7f783406d9d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-b con 0x7f78340b0860 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 10 monclient(hunting): picked mon.noname-c con 0x7f783413ef30 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:03.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78341dec10 con 0x7f783406d9d0 2024-03-29T10:37:03.025 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7834136ef0 con 0x7f78340b0860 2024-03-29T10:37:03.025 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f783259c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f783406d9d0 0x7f78340b3ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45770/0 (socket says 172.21.15.151:45770) 2024-03-29T10:37:03.025 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f783259c640 1 -- 172.21.15.151:0/3236518613 learned_addr learned my addr 172.21.15.151:0/3236518613 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:03.025 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78341db710 con 0x7f783413ef30 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7838be8640 10 monclient(hunting): authenticate will time out at 954.186874s 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7822ffd640 10 client.?.objecter ms_handle_connect 0x7f78340b0860 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7822ffd640 10 client.?.objecter ms_handle_connect 0x7f783406d9d0 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.017+0000 7f7822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975628926 0 0) 0x7f7834136ef0 con 0x7f78340b0860 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.026 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7814004eb0 con 0x7f78340b0860 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 client.?.objecter ms_handle_connect 0x7f783413ef30 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223197094 0 0) 0x7f78341dec10 con 0x7f783406d9d0 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7834136ef0 con 0x7f783406d9d0 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2308159410 0 0) 0x7f7814004eb0 con 0x7f78340b0860 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78341dec10 con 0x7f78340b0860 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.018+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f78280017d0 con 0x7f78340b0860 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): dump: 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.028 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742954874 0 0) 0x7f78341db710 con 0x7f783413ef30 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient(hunting): my global_id is 4226 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7814004eb0 con 0x7f783413ef30 2024-03-29T10:37:03.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3590165532 0 0) 0x7f7834136ef0 con 0x7f783406d9d0 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78341db710 con 0x7f783406d9d0 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470271518 0 0) 0x7f78341dec10 con 0x7f78340b0860 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 >> v1:172.21.15.151:6790/0 conn(0x7f783413ef30 legacy=0x7f78341dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 >> v1:172.21.15.151:6789/0 conn(0x7f783406d9d0 legacy=0x7f78340b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 monclient: found mon.b 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7834135f20 con 0x7f78340b0860 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient: _finish_auth 0 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:03.030 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 10 monclient: _renew_subs 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7834135580 con 0x7f78340b0860 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.019+0000 7f7838be8640 10 monclient: _renew_subs 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.020+0000 7f7838be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.020+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78341df990 con 0x7f78340b0860 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.020+0000 7f7838be8640 1 librados: init done 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7828001930 con 0x7f78340b0860 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f78280037b0 con 0x7f78340b0860 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:03.031 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7822ffd640 10 monclient: dump: 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:03.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7838be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7838be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7838be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7834075f10 con 0x7f78340b0860 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.021+0000 7f7823fff640 4 set_mon_vals no callback set 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.022+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f7828019780 con 0x7f78340b0860 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f783259c640 10 monclient: get_auth_request con 0x7f781406df30 auth_method 0 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f7828083630 con 0x7f78340b0860 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f7822ffd640 10 client.4234.objecter ms_dispatch 0x7f7834000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f7822ffd640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f7822ffd640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.023+0000 7f7822ffd640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.024+0000 7f7822ffd640 10 client.4234.objecter ms_handle_connect 0x7f781406df30 2024-03-29T10:37:03.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.028+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f782804ef50 con 0x7f78340b0860 2024-03-29T10:37:03.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.028+0000 7f7822ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:03.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.028+0000 7f7822ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:03.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.396+0000 7f7838be8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:03.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.396+0000 7f7838be8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:03.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.396+0000 7f7838be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:03.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.396+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7834072330 con 0x7f78340b0860 2024-03-29T10:37:03.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.397+0000 7f7822ffd640 1 -- 172.21.15.151:0/3236518613 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f7828050e80 con 0x7f78340b0860 2024-03-29T10:37:03.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.397+0000 7f7822ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:03.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.397+0000 7f7822ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:03.382 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:03.382 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:03.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 10 librados: watch_flush enter 2024-03-29T10:37:03.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 10 librados: watch_flush exit 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 >> v1:172.21.15.151:6816/1549873000 conn(0x7f781406df30 legacy=0x7f78140703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 10 monclient: shutdown 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:03.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 >> v1:172.21.15.169:6789/0 conn(0x7f78340b0860 legacy=0x7f78341df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:03.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 shutdown_connections 2024-03-29T10:37:03.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.400+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 >> 172.21.15.151:0/3236518613 conn(0x7f783406b660 msgr2=0x7f783413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:03.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.401+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 shutdown_connections 2024-03-29T10:37:03.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.401+0000 7f7838be8640 1 -- 172.21.15.151:0/3236518613 wait complete. 2024-03-29T10:37:03.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:03.401+0000 7f7838be8640 1 librados: shutdown 2024-03-29T10:37:03.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:10.209 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.371+0000 7f5de6ec1640 10 monclient: get_monmap_and_config 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.371+0000 7f5de6ec1640 10 monclient: build_initial_monmap 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.371+0000 7f5de6ec1640 10 monclient: monmap: 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:10.372218+0000 2024-03-29T10:37:10.356 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:10.372218+0000 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.372+0000 7f5de6ec1640 1 Processor -- start 2024-03-29T10:37:10.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.372+0000 7f5de6ec1640 1 -- start start 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.372+0000 7f5de6ec1640 10 monclient: init 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd8006a70 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd802b100 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd8029c30 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd802a270 con 0x7f5dd8029c30 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd802c440 con 0x7f5dd8006a70 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd80c9440 con 0x7f5dd802b100 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.373+0000 7f5de6ec1640 10 monclient(hunting): authenticate will time out at 961.543230s 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de5ebf640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5dd8006a70 0x7f5dd80c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:44096/0 (socket says 172.21.15.151:44096) 2024-03-29T10:37:10.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de5ebf640 1 -- 172.21.15.151:0/1147508053 learned_addr learned my addr 172.21.15.151:0/1147508053 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873610681 0 0) 0x7f5dd80c9440 con 0x7f5dd802b100 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc003520 con 0x7f5dd802b100 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652982927 0 0) 0x7f5dd802a270 con 0x7f5dd8029c30 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.374+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd80c9440 con 0x7f5dd8029c30 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402846140 0 0) 0x7f5dcc003520 con 0x7f5dd802b100 2024-03-29T10:37:10.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 >> v1:172.21.15.169:6789/0 conn(0x7f5dd8006a70 legacy=0x7f5dd80c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 >> v1:172.21.15.151:6789/0 conn(0x7f5dd8029c30 legacy=0x7f5dd80d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 monclient: found mon.noname-b 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80c95d0 con 0x7f5dd802b100 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: _finish_auth 0 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ddc001690 con 0x7f5dd802b100 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:10.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: dump: 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de6ec1640 5 monclient: authenticate success, global_id 4229 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de6ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ddc001860 con 0x7f5dd802b100 2024-03-29T10:37:10.364 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 1 -- 172.21.15.151:0/1147508053 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ddc001c70 con 0x7f5dd802b100 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de4ebd640 10 monclient: dump: 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.365 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de6ec1640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.375+0000 7f5de6ec1640 4 set_mon_vals no callback set 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 10 monclient: shutdown 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 1 -- 172.21.15.151:0/1147508053 >> v1:172.21.15.151:6790/0 conn(0x7f5dd802b100 legacy=0x7f5dd80cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 1 -- 172.21.15.151:0/1147508053 shutdown_connections 2024-03-29T10:37:10.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 1 -- 172.21.15.151:0/1147508053 >> 172.21.15.151:0/1147508053 conn(0x7f5dd80058f0 msgr2=0x7f5dd8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:10.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 1 -- 172.21.15.151:0/1147508053 shutdown_connections 2024-03-29T10:37:10.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.376+0000 7f5de6ec1640 1 -- 172.21.15.151:0/1147508053 wait complete. 2024-03-29T10:37:10.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.377+0000 7f5de6ec1640 10 monclient: build_initial_monmap 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.377+0000 7f5de6ec1640 10 monclient: monmap: 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 librados: starting msgr at 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 librados: starting objecter 2024-03-29T10:37:10.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 Processor -- start 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 -- start start 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 librados: setting wanted keys 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 1 librados: calling monclient init 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.378+0000 7f5de6ec1640 10 monclient: init 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd8006a70 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd8029c30 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd81566b0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de5ebf640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f5dd8006a70 0x7f5dd81556e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:54032/0 (socket says 172.21.15.151:54032) 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de5ebf640 1 -- 172.21.15.151:0/3381918105 learned_addr learned my addr 172.21.15.151:0/3381918105 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd8155df0 con 0x7f5dd8006a70 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.379+0000 7f5dd67fc640 10 client.?.objecter ms_handle_connect 0x7f5dd8006a70 2024-03-29T10:37:10.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd81583d0 con 0x7f5dd81566b0 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd8157510 con 0x7f5dd8029c30 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5de6ec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5de6ec1640 10 monclient(hunting): authenticate will time out at 961.549602s 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 client.?.objecter ms_handle_connect 0x7f5dd81566b0 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 client.?.objecter ms_handle_connect 0x7f5dd8029c30 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189923441 0 0) 0x7f5dd8155df0 con 0x7f5dd8006a70 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:10.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc4004eb0 con 0x7f5dd8006a70 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232172603 0 0) 0x7f5dd81583d0 con 0x7f5dd81566b0 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd8155df0 con 0x7f5dd81566b0 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886845348 0 0) 0x7f5dd8157510 con 0x7f5dd8029c30 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd81583d0 con 0x7f5dd8029c30 2024-03-29T10:37:10.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.380+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210626507 0 0) 0x7f5dd81583d0 con 0x7f5dd8029c30 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dd8157510 con 0x7f5dd8029c30 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879146629 0 0) 0x7f5dd8155df0 con 0x7f5dd81566b0 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dd81583d0 con 0x7f5dd81566b0 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374110549 0 0) 0x7f5dc4004eb0 con 0x7f5dd8006a70 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dd8155df0 con 0x7f5dd8006a70 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ddc0019d0 con 0x7f5dd8029c30 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): dump: 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.379 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5de004fdd0 con 0x7f5dd81566b0 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): dump: 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.380 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5dd00018b0 con 0x7f5dd8006a70 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient(hunting): dump: 2024-03-29T10:37:10.381 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3785922812 0 0) 0x7f5dd8157510 con 0x7f5dd8029c30 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 >> v1:172.21.15.169:6789/0 conn(0x7f5dd81566b0 legacy=0x7f5dd816b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 >> v1:172.21.15.151:6789/0 conn(0x7f5dd8006a70 legacy=0x7f5dd81556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:10.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 monclient: found mon.c 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd81577f0 con 0x7f5dd8029c30 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5de6ec1640 5 monclient: authenticate success, global_id 4232 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5de6ec1640 10 monclient: _renew_subs 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5de6ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd8158860 con 0x7f5dd8029c30 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ddc0036b0 con 0x7f5dd8029c30 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ddc0039c0 con 0x7f5dd8029c30 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:10.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd67fc640 10 monclient: dump: 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.381+0000 7f5dd77fe640 4 set_mon_vals no callback set 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.382+0000 7f5de6ec1640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:10.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.382+0000 7f5de6ec1640 10 monclient: _renew_subs 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.382+0000 7f5de6ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.382+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd81580d0 con 0x7f5dd8029c30 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.382+0000 7f5de6ec1640 1 librados: init done 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.383+0000 7f5de6ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.383+0000 7f5de6ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.383+0000 7f5de6ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.383+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd8004cd0 con 0x7f5dd8029c30 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.384+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f5ddc019990 con 0x7f5dd8029c30 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f5ddc083720 con 0x7f5dd8029c30 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 10 client.4232.objecter ms_dispatch 0x7f5de0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5de5ebf640 10 monclient: get_auth_request con 0x7f5dc406e0c0 auth_method 0 2024-03-29T10:37:10.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.385+0000 7f5dd67fc640 10 client.4232.objecter ms_handle_connect 0x7f5dc406e0c0 2024-03-29T10:37:10.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.393+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ddc04f000 con 0x7f5dd8029c30 2024-03-29T10:37:10.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.393+0000 7f5dd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:10.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.393+0000 7f5dd67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:10.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.747+0000 7f5de6ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.747+0000 7f5de6ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.747+0000 7f5de6ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.747+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dd80086f0 con 0x7f5dd8029c30 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.748+0000 7f5dd67fc640 1 -- 172.21.15.151:0/3381918105 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f5ddc050f30 con 0x7f5dd8029c30 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.748+0000 7f5dd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:10.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.748+0000 7f5dd67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:10.734 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:10.734 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:10.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 10 librados: watch_flush enter 2024-03-29T10:37:10.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 10 librados: watch_flush exit 2024-03-29T10:37:10.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-29T10:37:10.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 10 client.4232.objecter successfully canceled tick 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 >> v1:172.21.15.151:6816/1549873000 conn(0x7f5dc406e0c0 legacy=0x7f5dc4070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 10 monclient: shutdown 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 >> v1:172.21.15.151:6790/0 conn(0x7f5dd8029c30 legacy=0x7f5dd8155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.752+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 shutdown_connections 2024-03-29T10:37:10.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.753+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 >> 172.21.15.151:0/3381918105 conn(0x7f5dd80058f0 msgr2=0x7f5dd8034010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:10.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.753+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 shutdown_connections 2024-03-29T10:37:10.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.753+0000 7f5de6ec1640 1 -- 172.21.15.151:0/3381918105 wait complete. 2024-03-29T10:37:10.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:10.753+0000 7f5de6ec1640 1 librados: shutdown 2024-03-29T10:37:10.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:17.559 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.724+0000 7f4d2eacf640 10 monclient: get_monmap_and_config 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.724+0000 7f4d2eacf640 10 monclient: build_initial_monmap 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.724+0000 7f4d2eacf640 10 monclient: monmap: 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:17.725604+0000 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:17.725604+0000 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.709 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:17.710 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:17.710 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:17.710 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.725+0000 7f4d2eacf640 1 Processor -- start 2024-03-29T10:37:17.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.725+0000 7f4d2eacf640 1 -- start start 2024-03-29T10:37:17.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.725+0000 7f4d2eacf640 10 monclient: init 2024-03-29T10:37:17.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-b con 0x7f4d28070170 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-c con 0x7f4d280713f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-a con 0x7f4d28070550 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2813dbd0 con 0x7f4d28070550 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2813ff70 con 0x7f4d280713f0 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d28140250 con 0x7f4d28070170 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.726+0000 7f4d2eacf640 10 monclient(hunting): authenticate will time out at 968.896426s 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.727+0000 7f4d27fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4d280713f0 0x7f4d281406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:33906/0 (socket says 172.21.15.151:33906) 2024-03-29T10:37:17.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.727+0000 7f4d27fff640 1 -- 172.21.15.151:0/3225369227 learned_addr learned my addr 172.21.15.151:0/3225369227 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926393789 0 0) 0x7f4d2813ff70 con 0x7f4d280713f0 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.728+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18003520 con 0x7f4d280713f0 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.729+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902887243 0 0) 0x7f4d18003520 con 0x7f4d280713f0 2024-03-29T10:37:17.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.729+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 >> v1:172.21.15.151:6790/0 conn(0x7f4d28070170 legacy=0x7f4d2813d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 >> v1:172.21.15.151:6789/0 conn(0x7f4d28070550 legacy=0x7f4d2814b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 1 monclient: found mon.noname-c 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28140df0 con 0x7f4d280713f0 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.731+0000 7f4d277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d2eacf640 5 monclient: authenticate success, global_id 4243 2024-03-29T10:37:17.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d2eacf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d1c001690 con 0x7f4d280713f0 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d277fe640 10 monclient: dump: 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:17.718 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.732+0000 7f4d2eacf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d1c0018f0 con 0x7f4d280713f0 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 1 -- 172.21.15.151:0/3225369227 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d1c001d40 con 0x7f4d280713f0 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.733+0000 7f4d277fe640 10 monclient: dump: 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.721 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.734+0000 7f4d2eacf640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.734+0000 7f4d2eacf640 4 set_mon_vals no callback set 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.735+0000 7f4d2eacf640 10 monclient: shutdown 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.735+0000 7f4d2eacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:17.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.735+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3225369227 >> v1:172.21.15.169:6789/0 conn(0x7f4d280713f0 legacy=0x7f4d281406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:17.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.737+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3225369227 shutdown_connections 2024-03-29T10:37:17.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.737+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3225369227 >> 172.21.15.151:0/3225369227 conn(0x7f4d28067610 msgr2=0x7f4d280b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:17.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.738+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3225369227 shutdown_connections 2024-03-29T10:37:17.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.738+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3225369227 wait complete. 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.739+0000 7f4d2eacf640 10 monclient: build_initial_monmap 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.739+0000 7f4d2eacf640 10 monclient: monmap: 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:17.726 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.739+0000 7f4d2eacf640 1 librados: starting msgr at 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.739+0000 7f4d2eacf640 1 librados: starting objecter 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.739+0000 7f4d2eacf640 1 Processor -- start 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 1 -- start start 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 1 librados: setting wanted keys 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 1 librados: calling monclient init 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 10 monclient: init 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:17.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-b con 0x7f4d28070170 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.740+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-c con 0x7f4d28070550 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 10 monclient(hunting): picked mon.noname-a con 0x7f4d280713f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d281360b0 con 0x7f4d280713f0 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d281342d0 con 0x7f4d28070170 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d281d54b0 con 0x7f4d28070550 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:17.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.741+0000 7f4d2eacf640 10 monclient(hunting): authenticate will time out at 968.910552s 2024-03-29T10:37:17.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.744+0000 7f4d2d06c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4d280713f0 0x7f4d281df7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:54021/0 (socket says 172.21.15.151:54021) 2024-03-29T10:37:17.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.744+0000 7f4d2d06c640 1 -- 172.21.15.151:0/3122590735 learned_addr learned my addr 172.21.15.151:0/3122590735 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:17.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 client.?.objecter ms_handle_connect 0x7f4d280713f0 2024-03-29T10:37:17.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:17.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 client.?.objecter ms_handle_connect 0x7f4d28070550 2024-03-29T10:37:17.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946648110 0 0) 0x7f4d281d54b0 con 0x7f4d28070550 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c0092c0 con 0x7f4d28070550 2024-03-29T10:37:17.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732082382 0 0) 0x7f4d281360b0 con 0x7f4d280713f0 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d281d54b0 con 0x7f4d280713f0 2024-03-29T10:37:17.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463566746 0 0) 0x7f4d0c0092c0 con 0x7f4d28070550 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d281360b0 con 0x7f4d28070550 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.745+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d1c0014b0 con 0x7f4d28070550 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient(hunting): dump: 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:17.740 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1835303104 0 0) 0x7f4d281360b0 con 0x7f4d28070550 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 >> v1:172.21.15.169:6789/0 conn(0x7f4d28070170 legacy=0x7f4d28135650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 >> v1:172.21.15.151:6789/0 conn(0x7f4d280713f0 legacy=0x7f4d281df7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: _finish_hunting 0 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 monclient: found mon.c 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28137760 con 0x7f4d28070550 2024-03-29T10:37:17.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: _finish_auth 0 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: _check_auth_tickets 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 5 monclient: authenticate success, global_id 4235 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 10 monclient: _renew_subs 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d280a9580 con 0x7f4d28070550 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 10 monclient: _renew_subs 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2814c500 con 0x7f4d28070550 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d2eacf640 1 librados: init done 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d1c001610 con 0x7f4d28070550 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:17.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d1c004e10 con 0x7f4d28070550 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.746+0000 7f4d257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.747+0000 7f4d257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.747+0000 7f4d267fc640 4 set_mon_vals no callback set 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.747+0000 7f4d257fa640 10 monclient: dump: 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:17.743 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.748+0000 7f4d2eacf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.748+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4d1c01ade0 con 0x7f4d28070550 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.748+0000 7f4d2eacf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.748+0000 7f4d2eacf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.748+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d08005080 con 0x7f4d28070550 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4d1c054910 con 0x7f4d28070550 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d257fa640 10 client.4235.objecter ms_dispatch 0x7f4d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d257fa640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d257fa640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d257fa640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.749+0000 7f4d2c86b640 10 monclient: get_auth_request con 0x7f4d0c0720b0 auth_method 0 2024-03-29T10:37:17.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.750+0000 7f4d257fa640 10 client.4235.objecter ms_handle_connect 0x7f4d0c0720b0 2024-03-29T10:37:17.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.758+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d1c050bb0 con 0x7f4d28070550 2024-03-29T10:37:17.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.758+0000 7f4d257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:17.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:17.758+0000 7f4d257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:18.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.082+0000 7f4d2eacf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:18.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.082+0000 7f4d2eacf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:18.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.082+0000 7f4d2eacf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:18.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.082+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d08005330 con 0x7f4d28070550 2024-03-29T10:37:18.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.083+0000 7f4d257fa640 1 -- 172.21.15.151:0/3122590735 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4d1c050d50 con 0x7f4d28070550 2024-03-29T10:37:18.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.083+0000 7f4d257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:18.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.083+0000 7f4d257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:18.068 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:18.068 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 10 librados: watch_flush enter 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 10 librados: watch_flush exit 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 10 client.4235.objecter successfully canceled tick 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4d0c0720b0 legacy=0x7f4d0c074570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 10 monclient: shutdown 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 >> v1:172.21.15.151:6790/0 conn(0x7f4d28070550 legacy=0x7f4d281d4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 shutdown_connections 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 >> 172.21.15.151:0/3122590735 conn(0x7f4d28067610 msgr2=0x7f4d28069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 shutdown_connections 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 -- 172.21.15.151:0/3122590735 wait complete. 2024-03-29T10:37:18.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:18.087+0000 7f4d2eacf640 1 librados: shutdown 2024-03-29T10:37:18.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:24.895 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.058+0000 7f3cbbfa4640 10 monclient: get_monmap_and_config 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.058+0000 7f3cbbfa4640 10 monclient: build_initial_monmap 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.058+0000 7f3cbbfa4640 10 monclient: monmap: 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:25.043 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:25.059780+0000 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:25.059780+0000 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:25.044 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.059+0000 7f3cbbfa4640 1 Processor -- start 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.060+0000 7f3cbbfa4640 1 -- start start 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.060+0000 7f3cbbfa4640 10 monclient: init 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.061+0000 7f3cbbfa4640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.061+0000 7f3cbbfa4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.061+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-a con 0x7f3cac006800 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:25.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.061+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-b con 0x7f3cac006be0 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:25.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cac031b60 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:25.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac029370 con 0x7f3cac006800 2024-03-29T10:37:25.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0295e0 con 0x7f3cac031b60 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac031360 con 0x7f3cac006be0 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cbbfa4640 10 monclient(hunting): authenticate will time out at 976.233704s 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cba541640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3cac031b60 0x7f3cac0d30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:40323/0 (socket says 172.21.15.151:40323) 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.064+0000 7f3cba541640 1 -- 172.21.15.151:0/498490461 learned_addr learned my addr 172.21.15.151:0/498490461 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562946901 0 0) 0x7f3cac0295e0 con 0x7f3cac031b60 2024-03-29T10:37:25.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 10 monclient(hunting): _negotiate 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca0003520 con 0x7f3cac031b60 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943108637 0 0) 0x7f3ca0003520 con 0x7f3cac031b60 2024-03-29T10:37:25.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.065+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 >> v1:172.21.15.151:6790/0 conn(0x7f3cac006be0 legacy=0x7f3cac0c4ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:25.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 >> v1:172.21.15.151:6789/0 conn(0x7f3cac006800 legacy=0x7f3cac0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:25.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: _finish_hunting 0 2024-03-29T10:37:25.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 1 monclient: found mon.noname-c 2024-03-29T10:37:25.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac031640 con 0x7f3cac031b60 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: _finish_auth 0 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: _check_auth_tickets 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3cb404f730 con 0x7f3cac031b60 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.067+0000 7f3cb8d3e640 10 monclient: dump: 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:25.053 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:25.054 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb4075f10 con 0x7f3cac031b60 2024-03-29T10:37:25.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:25.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 1 -- 172.21.15.151:0/498490461 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3cb40758b0 con 0x7f3cac031b60 2024-03-29T10:37:25.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:25.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cb8d3e640 10 monclient: dump: 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:25.056 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cbbfa4640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cbbfa4640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.069+0000 7f3cbbfa4640 4 set_mon_vals no callback set 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.070+0000 7f3cbbfa4640 10 monclient: shutdown 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.070+0000 7f3cbbfa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.070+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/498490461 >> v1:172.21.15.169:6789/0 conn(0x7f3cac031b60 legacy=0x7f3cac0d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.071+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/498490461 shutdown_connections 2024-03-29T10:37:25.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.071+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/498490461 >> 172.21.15.151:0/498490461 conn(0x7f3cac0058f0 msgr2=0x7f3cac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:25.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.072+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/498490461 shutdown_connections 2024-03-29T10:37:25.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.074+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/498490461 wait complete. 2024-03-29T10:37:25.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.075+0000 7f3cbbfa4640 10 monclient: build_initial_monmap 2024-03-29T10:37:25.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.075+0000 7f3cbbfa4640 10 monclient: monmap: 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.076+0000 7f3cbbfa4640 1 librados: starting msgr at 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.076+0000 7f3cbbfa4640 1 librados: starting objecter 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.076+0000 7f3cbbfa4640 1 Processor -- start 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.076+0000 7f3cbbfa4640 1 -- start start 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.076+0000 7f3cbbfa4640 1 librados: setting wanted keys 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.077+0000 7f3cbbfa4640 1 librados: calling monclient init 2024-03-29T10:37:25.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.077+0000 7f3cbbfa4640 10 monclient: init 2024-03-29T10:37:25.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.079+0000 7f3cbbfa4640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:25.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.079+0000 7f3cbbfa4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:37:25.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.079+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cac006800 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:25.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.079+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-a con 0x7f3cac031b60 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 10 monclient(hunting): picked mon.noname-b con 0x7f3cac0c55a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0c6d70 con 0x7f3cac031b60 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0bd7e0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0c3bb0 con 0x7f3cac006800 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:25.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.081+0000 7f3cbbfa4640 10 monclient(hunting): authenticate will time out at 976.251467s 2024-03-29T10:37:25.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.083+0000 7f3cba541640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3cac0c55a0 0x7f3cac15c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:40419/0 (socket says 172.21.15.151:40419) 2024-03-29T10:37:25.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.083+0000 7f3cba541640 1 -- 172.21.15.151:0/1276910413 learned_addr learned my addr 172.21.15.151:0/1276910413 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 client.?.objecter ms_handle_connect 0x7f3cac0c55a0 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 client.?.objecter ms_handle_connect 0x7f3cac031b60 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028804866 0 0) 0x7f3cac0bd7e0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c98004eb0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291489776 0 0) 0x7f3cac0c6d70 con 0x7f3cac031b60 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:25.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.086+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac0bd7e0 con 0x7f3cac031b60 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749402390 0 0) 0x7f3c98004eb0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cac0c6d70 con 0x7f3cac0c55a0 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3cb404f400 con 0x7f3cac0c55a0 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient(hunting): dump: 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.074 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478419660 0 0) 0x7f3cac0bd7e0 con 0x7f3cac031b60 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c98004eb0 con 0x7f3cac031b60 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1193168330 0 0) 0x7f3cac0c6d70 con 0x7f3cac0c55a0 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 >> v1:172.21.15.151:6790/0 conn(0x7f3cac006800 legacy=0x7f3cac0c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 >> v1:172.21.15.151:6789/0 conn(0x7f3cac031b60 legacy=0x7f3cac0bd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:25.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 monclient: found mon.b 2024-03-29T10:37:25.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:25.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0c3d40 con 0x7f3cac0c55a0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 10 monclient: _renew_subs 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac15cac0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 10 monclient: _renew_subs 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac0ca560 con 0x7f3cac0c55a0 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3cbbfa4640 1 librados: init done 2024-03-29T10:37:25.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb404f730 con 0x7f3cac0c55a0 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3cb4067bf0 con 0x7f3cac0c55a0 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.087+0000 7f3caa7fc640 10 monclient: dump: 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:25.079 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.089+0000 7f3cabfff640 4 set_mon_vals no callback set 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.090+0000 7f3cbbfa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.090+0000 7f3cbbfa4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.090+0000 7f3cbbfa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.093+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f3cb4067d50 con 0x7f3cac0c55a0 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f3cb40ff340 con 0x7f3cac0c55a0 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 10 client.4249.objecter ms_dispatch 0x7f3cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.090+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c005080 con 0x7f3cac0c55a0 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3cb9d40640 10 monclient: get_auth_request con 0x7f3c9806dcd0 auth_method 0 2024-03-29T10:37:25.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.094+0000 7f3caa7fc640 10 client.4249.objecter ms_handle_connect 0x7f3c9806dcd0 2024-03-29T10:37:25.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.106+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3cb40cac20 con 0x7f3cac0c55a0 2024-03-29T10:37:25.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.106+0000 7f3caa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:25.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.106+0000 7f3caa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.433+0000 7f3cbbfa4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.433+0000 7f3cbbfa4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.433+0000 7f3cbbfa4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.433+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c9c005330 con 0x7f3cac0c55a0 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.436+0000 7f3caa7fc640 1 -- 172.21.15.151:0/1276910413 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f3cb40ccb50 con 0x7f3cac0c55a0 2024-03-29T10:37:25.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.436+0000 7f3caa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:25.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.436+0000 7f3caa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:25.421 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:25.422 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:25.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.441+0000 7f3cbbfa4640 10 librados: watch_flush enter 2024-03-29T10:37:25.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.441+0000 7f3cbbfa4640 10 librados: watch_flush exit 2024-03-29T10:37:25.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.441+0000 7f3cbbfa4640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T10:37:25.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.441+0000 7f3cbbfa4640 10 client.4249.objecter successfully canceled tick 2024-03-29T10:37:25.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.441+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 >> v1:172.21.15.151:6816/1549873000 conn(0x7f3c9806dcd0 legacy=0x7f3c98070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:25.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.442+0000 7f3cbbfa4640 10 monclient: shutdown 2024-03-29T10:37:25.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.442+0000 7f3cbbfa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:25.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.442+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 >> v1:172.21.15.169:6789/0 conn(0x7f3cac0c55a0 legacy=0x7f3cac15c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:25.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.442+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 shutdown_connections 2024-03-29T10:37:25.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.442+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 >> 172.21.15.151:0/1276910413 conn(0x7f3cac0058f0 msgr2=0x7f3cac02a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:25.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.444+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 shutdown_connections 2024-03-29T10:37:25.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.444+0000 7f3cbbfa4640 1 -- 172.21.15.151:0/1276910413 wait complete. 2024-03-29T10:37:25.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:25.445+0000 7f3cbbfa4640 1 librados: shutdown 2024-03-29T10:37:25.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:32.261 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:32.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.430+0000 7fe5edf1c640 10 monclient: get_monmap_and_config 2024-03-29T10:37:32.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.430+0000 7fe5edf1c640 10 monclient: build_initial_monmap 2024-03-29T10:37:32.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.430+0000 7fe5edf1c640 10 monclient: monmap: 2024-03-29T10:37:32.415 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:32.431811+0000 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:32.431811+0000 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:32.416 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.432+0000 7fe5edf1c640 1 Processor -- start 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.432+0000 7fe5edf1c640 1 -- start start 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.432+0000 7fe5edf1c640 10 monclient: init 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e80b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e814a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e80ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:32.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80aba70 con 0x7fe5e80b0770 2024-03-29T10:37:32.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80aecb0 con 0x7fe5e814a700 2024-03-29T10:37:32.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80af150 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:32.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.433+0000 7fe5edf1c640 10 monclient(hunting): authenticate will time out at 983.602622s 2024-03-29T10:37:32.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.439+0000 7fe5e6ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe5e814a700 0x7fe5e80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:56077/0 (socket says 172.21.15.151:56077) 2024-03-29T10:37:32.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.439+0000 7fe5e6ffd640 1 -- 172.21.15.151:0/2841495477 learned_addr learned my addr 172.21.15.151:0/2841495477 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386749734 0 0) 0x7fe5e80aecb0 con 0x7fe5e814a700 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:37:32.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.440+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d4003520 con 0x7fe5e814a700 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529515692 0 0) 0x7fe5d4003520 con 0x7fe5e814a700 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 >> v1:172.21.15.151:6790/0 conn(0x7fe5e80ac5a0 legacy=0x7fe5e8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 >> v1:172.21.15.151:6789/0 conn(0x7fe5e80b0770 legacy=0x7fe5e8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 monclient: found mon.noname-c 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80ac9d0 con 0x7fe5e814a700 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:32.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe5e00016b0 con 0x7fe5e814a700 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: dump: 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:32.429 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5e0001880 con 0x7fe5e814a700 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 1 -- 172.21.15.151:0/2841495477 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe5e0001cd0 con 0x7fe5e814a700 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.442+0000 7fe5e67fc640 10 monclient: dump: 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:32.430 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.443+0000 7fe5edf1c640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.443+0000 7fe5edf1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:32.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.443+0000 7fe5edf1c640 4 set_mon_vals no callback set 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.446+0000 7fe5edf1c640 10 monclient: shutdown 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.446+0000 7fe5edf1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.446+0000 7fe5edf1c640 1 -- 172.21.15.151:0/2841495477 >> v1:172.21.15.169:6789/0 conn(0x7fe5e814a700 legacy=0x7fe5e80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.447+0000 7fe5edf1c640 1 -- 172.21.15.151:0/2841495477 shutdown_connections 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.447+0000 7fe5edf1c640 1 -- 172.21.15.151:0/2841495477 >> 172.21.15.151:0/2841495477 conn(0x7fe5e806b660 msgr2=0x7fe5e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.447+0000 7fe5edf1c640 1 -- 172.21.15.151:0/2841495477 shutdown_connections 2024-03-29T10:37:32.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.447+0000 7fe5edf1c640 1 -- 172.21.15.151:0/2841495477 wait complete. 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.448+0000 7fe5edf1c640 10 monclient: build_initial_monmap 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.448+0000 7fe5edf1c640 10 monclient: monmap: 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:32.434 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.448+0000 7fe5edf1c640 1 librados: starting msgr at 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.448+0000 7fe5edf1c640 1 librados: starting objecter 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.448+0000 7fe5edf1c640 1 Processor -- start 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 1 -- start start 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 1 librados: setting wanted keys 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 1 librados: calling monclient init 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 10 monclient: init 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:32.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.449+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e80ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e80b0770 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e814a700 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8128120 con 0x7fe5e814a700 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8147b90 con 0x7fe5e80b0770 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e8127d80 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:32.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.450+0000 7fe5edf1c640 10 monclient(hunting): authenticate will time out at 983.619582s 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5e77fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe5e80ac5a0 0x7fe5e8128640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:39865/0 (socket says 172.21.15.151:39865) 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5e77fe640 1 -- 172.21.15.151:0/297053103 learned_addr learned my addr 172.21.15.151:0/297053103 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5dffff640 10 client.?.objecter ms_handle_connect 0x7fe5e80ac5a0 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225820139 0 0) 0x7fe5e8127d80 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.454+0000 7fe5dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c8004eb0 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 client.?.objecter ms_handle_connect 0x7fe5e80b0770 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214432021 0 0) 0x7fe5e8147b90 con 0x7fe5e80b0770 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e8127d80 con 0x7fe5e80b0770 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958143826 0 0) 0x7fe5c8004eb0 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.455+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e8147b90 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe5d0004000 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:32.442 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 10 monclient(hunting): dump: 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1812920113 0 0) 0x7fe5e8147b90 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 >> v1:172.21.15.169:6789/0 conn(0x7fe5e80b0770 legacy=0x7fe5e8128d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:32.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.456+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 >> v1:172.21.15.151:6789/0 conn(0x7fe5e814a700 legacy=0x7fe5e81e5fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.458+0000 7fe5dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.458+0000 7fe5dffff640 1 monclient: found mon.c 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e81df680 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 10 monclient: _finish_auth 0 2024-03-29T10:37:32.446 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5d0000ed0 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 10 monclient: _renew_subs 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e81df930 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 10 monclient: _renew_subs 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e81dedf0 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5edf1c640 1 librados: init done 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.459+0000 7fe5e5ffb640 4 set_mon_vals no callback set 2024-03-29T10:37:32.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.460+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe5d00012e0 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.462+0000 7fe5dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.462+0000 7fe5dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.462+0000 7fe5dffff640 10 monclient: dump: 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:32.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.462+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe5d0001440 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.449 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe5d0084870 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5dffff640 10 client.4238.objecter ms_dispatch 0x7fe5e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5dffff640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5dffff640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5dffff640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.463+0000 7fe5e6ffd640 10 monclient: get_auth_request con 0x7fe5c806dc80 auth_method 0 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.464+0000 7fe5dffff640 10 client.4238.objecter ms_handle_connect 0x7fe5c806dc80 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.466+0000 7fe5edf1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.466+0000 7fe5edf1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:32.450 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.466+0000 7fe5edf1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:32.451 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.466+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc005080 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.481+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5d0050150 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.481+0000 7fe5dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:32.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.481+0000 7fe5dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:32.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.802+0000 7fe5edf1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:32.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.802+0000 7fe5edf1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:32.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.802+0000 7fe5edf1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:37:32.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.802+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5cc005330 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.803+0000 7fe5dffff640 1 -- 172.21.15.151:0/297053103 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe5d0052080 con 0x7fe5e80ac5a0 2024-03-29T10:37:32.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.803+0000 7fe5dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:32.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.803+0000 7fe5dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:32.788 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:32.788 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:32.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 10 librados: watch_flush enter 2024-03-29T10:37:32.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 10 librados: watch_flush exit 2024-03-29T10:37:32.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 10 client.4238.objecter successfully canceled tick 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe5c806dc80 legacy=0x7fe5c8070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 10 monclient: shutdown 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 >> v1:172.21.15.151:6790/0 conn(0x7fe5e80ac5a0 legacy=0x7fe5e8128640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 shutdown_connections 2024-03-29T10:37:32.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.807+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 >> 172.21.15.151:0/297053103 conn(0x7fe5e806b660 msgr2=0x7fe5e806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:32.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.809+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 shutdown_connections 2024-03-29T10:37:32.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.809+0000 7fe5edf1c640 1 -- 172.21.15.151:0/297053103 wait complete. 2024-03-29T10:37:32.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:32.810+0000 7fe5edf1c640 1 librados: shutdown 2024-03-29T10:37:32.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:39.619 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:39.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.786+0000 7f39ae83b640 10 monclient: get_monmap_and_config 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.786+0000 7f39ae83b640 10 monclient: build_initial_monmap 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.786+0000 7f39ae83b640 10 monclient: monmap: 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:39.787400+0000 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:39.787400+0000 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:39.771 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.787+0000 7f39ae83b640 1 Processor -- start 2024-03-29T10:37:39.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.787+0000 7f39ae83b640 1 -- start start 2024-03-29T10:37:39.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.787+0000 7f39ae83b640 10 monclient: init 2024-03-29T10:37:39.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:39.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:37:39.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-a con 0x7f39a0036a60 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:39.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-c con 0x7f39a00c5de0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:39.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-b con 0x7f39a00c3070 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a002cc60 con 0x7f39a0036a60 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a002e040 con 0x7f39a00c5de0 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a002e320 con 0x7f39a00c3070 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.788+0000 7f39ae83b640 10 monclient(hunting): authenticate will time out at 990.958355s 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.793+0000 7f39ad038640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f39a00c5de0 0x7f39a00c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:44805/0 (socket says 172.21.15.151:44805) 2024-03-29T10:37:39.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.793+0000 7f39ad038640 1 -- 172.21.15.151:0/968322083 learned_addr learned my addr 172.21.15.151:0/968322083 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:39.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445929274 0 0) 0x7f39a002e040 con 0x7f39a00c5de0 2024-03-29T10:37:39.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 10 monclient(hunting): _negotiate 2024-03-29T10:37:39.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:39.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:39.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:37:39.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.794+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994003520 con 0x7f39a00c5de0 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.795+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868002741 0 0) 0x7f39a002e320 con 0x7f39a00c3070 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.795+0000 7f39ac837640 10 monclient(hunting): _negotiate 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.795+0000 7f39ac837640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.795+0000 7f39ac837640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.796+0000 7f39ac837640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:37:39.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.796+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a002e040 con 0x7f39a00c3070 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218387043 0 0) 0x7f3994003520 con 0x7f39a00c5de0 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 >> v1:172.21.15.151:6790/0 conn(0x7f39a00c3070 legacy=0x7f39a00d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 >> v1:172.21.15.151:6789/0 conn(0x7f39a0036a60 legacy=0x7f39a00c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: _finish_hunting 0 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 monclient: found mon.noname-c 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a002c830 con 0x7f39a00c5de0 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: _finish_auth 0 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: _check_auth_tickets 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:39.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3998001690 con 0x7f39a00c5de0 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ac837640 10 monclient: dump: 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ae83b640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.799+0000 7f39ae83b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3998001860 con 0x7f39a00c5de0 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 1 -- 172.21.15.151:0/968322083 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3998001c70 con 0x7f39a00c5de0 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ac837640 10 monclient: dump: 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:39.787 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:39.788 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ae83b640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:39.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.800+0000 7f39ae83b640 4 set_mon_vals no callback set 2024-03-29T10:37:39.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.801+0000 7f39ae83b640 10 monclient: shutdown 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.801+0000 7f39ae83b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.801+0000 7f39ae83b640 1 -- 172.21.15.151:0/968322083 >> v1:172.21.15.169:6789/0 conn(0x7f39a00c5de0 legacy=0x7f39a00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.802+0000 7f39ae83b640 1 -- 172.21.15.151:0/968322083 shutdown_connections 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.802+0000 7f39ae83b640 1 -- 172.21.15.151:0/968322083 >> 172.21.15.151:0/968322083 conn(0x7f39a0005c00 msgr2=0x7f39a0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.802+0000 7f39ae83b640 1 -- 172.21.15.151:0/968322083 shutdown_connections 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.802+0000 7f39ae83b640 1 -- 172.21.15.151:0/968322083 wait complete. 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.803+0000 7f39ae83b640 10 monclient: build_initial_monmap 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.803+0000 7f39ae83b640 10 monclient: monmap: 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:39.790 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 librados: starting msgr at 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 librados: starting objecter 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 Processor -- start 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 -- start start 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 librados: setting wanted keys 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 1 librados: calling monclient init 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.804+0000 7f39ae83b640 10 monclient: init 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.805+0000 7f39ae83b640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:39.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.805+0000 7f39ae83b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:39.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.807+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-a con 0x7f39a0036a60 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:39.793 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-c con 0x7f39a00c3070 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 10 monclient(hunting): picked mon.noname-b con 0x7f39a00c84f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a00c9230 con 0x7f39a0036a60 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a00c4b90 con 0x7f39a00c84f0 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a00c4d20 con 0x7f39a00c3070 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.808+0000 7f39ae83b640 10 monclient(hunting): authenticate will time out at 990.977605s 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.810+0000 7f39ad839640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f39a0036a60 0x7f39a00bc010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35321/0 (socket says 172.21.15.151:35321) 2024-03-29T10:37:39.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.810+0000 7f39ad839640 1 -- 172.21.15.151:0/2104853466 learned_addr learned my addr 172.21.15.151:0/2104853466 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 client.?.objecter ms_handle_connect 0x7f39a0036a60 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289622052 0 0) 0x7f39a00c9230 con 0x7f39a0036a60 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.812+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c004eb0 con 0x7f39a0036a60 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 client.?.objecter ms_handle_connect 0x7f39a00c84f0 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:39.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834293363 0 0) 0x7f39a00c4b90 con 0x7f39a00c84f0 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a00c9230 con 0x7f39a00c84f0 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209471103 0 0) 0x7f398c004eb0 con 0x7f39a0036a60 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a00c4b90 con 0x7f39a0036a60 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 client.?.objecter ms_handle_connect 0x7f39a00c3070 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f39a4004080 con 0x7f39a0036a60 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.813+0000 7f399e7fc640 10 monclient(hunting): dump: 2024-03-29T10:37:39.802 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304984844 0 0) 0x7f39a00c9230 con 0x7f39a00c84f0 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f398c004eb0 con 0x7f39a00c84f0 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304643790 0 0) 0x7f39a00c4b90 con 0x7f39a0036a60 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 >> v1:172.21.15.151:6790/0 conn(0x7f39a00c3070 legacy=0x7f39a015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:39.803 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 >> v1:172.21.15.169:6789/0 conn(0x7f39a00c84f0 legacy=0x7f39a015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 monclient: found mon.a 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a00c4eb0 con 0x7f39a0036a60 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f399e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f39ae83b640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.814+0000 7f39ae83b640 10 monclient: _renew_subs 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:39.804 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a00bc720 con 0x7f39a0036a60 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 10 monclient: _renew_subs 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a00c8ad0 con 0x7f39a0036a60 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.815+0000 7f39ae83b640 1 librados: init done 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a40044d0 con 0x7f39a0036a60 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f39a4001180 con 0x7f39a0036a60 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399e7fc640 10 monclient: dump: 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:39.805 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:39.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.816+0000 7f399ffff640 4 set_mon_vals no callback set 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.819+0000 7f39ae83b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.819+0000 7f39ae83b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.819+0000 7f39ae83b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.819+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990005080 con 0x7f39a0036a60 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.820+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f39a40012e0 con 0x7f39a0036a60 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f39ad038640 10 monclient: get_auth_request con 0x7f398c06dd50 auth_method 0 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f39a4084a70 con 0x7f39a0036a60 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 10 client.4257.objecter ms_dispatch 0x7f39a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:39.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:39.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:39.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:37:39.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.821+0000 7f399e7fc640 10 client.4257.objecter ms_handle_connect 0x7f398c06dd50 2024-03-29T10:37:39.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.829+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a4050180 con 0x7f39a0036a60 2024-03-29T10:37:39.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.829+0000 7f399e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:39.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:39.829+0000 7f399e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:40.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.162+0000 7f39ae83b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:40.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.162+0000 7f39ae83b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:40.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.162+0000 7f39ae83b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:40.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.162+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3990005330 con 0x7f39a0036a60 2024-03-29T10:37:40.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.165+0000 7f399e7fc640 1 -- 172.21.15.151:0/2104853466 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f39a4052280 con 0x7f39a0036a60 2024-03-29T10:37:40.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.165+0000 7f399e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:40.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.165+0000 7f399e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:40.151 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:40.152 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:40.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.171+0000 7f39ae83b640 10 librados: watch_flush enter 2024-03-29T10:37:40.156 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.171+0000 7f39ae83b640 10 librados: watch_flush exit 2024-03-29T10:37:40.156 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.171+0000 7f39ae83b640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:37:40.156 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.171+0000 7f39ae83b640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 >> v1:172.21.15.151:6816/1549873000 conn(0x7f398c06dd50 legacy=0x7f398c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 10 monclient: shutdown 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 >> v1:172.21.15.151:6789/0 conn(0x7f39a0036a60 legacy=0x7f39a00bc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 shutdown_connections 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 >> 172.21.15.151:0/2104853466 conn(0x7f39a0005c00 msgr2=0x7f39a00bf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 shutdown_connections 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 -- 172.21.15.151:0/2104853466 wait complete. 2024-03-29T10:37:40.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:40.172+0000 7f39ae83b640 1 librados: shutdown 2024-03-29T10:37:40.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:46.980 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.145+0000 7fd3c2c8d640 10 monclient: get_monmap_and_config 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.145+0000 7fd3c2c8d640 10 monclient: build_initial_monmap 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.145+0000 7fd3c2c8d640 10 monclient: monmap: 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:47.146688+0000 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:47.146688+0000 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.130 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.146+0000 7fd3c2c8d640 1 Processor -- start 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.147+0000 7fd3c2c8d640 1 -- start start 2024-03-29T10:37:47.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.147+0000 7fd3c2c8d640 10 monclient: init 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd3bc1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd3bc13f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd3bc0a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc0aaef0 con 0x7fd3bc0a9d70 2024-03-29T10:37:47.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc145bd0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc145eb0 con 0x7fd3bc13f650 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.148+0000 7fd3c2c8d640 10 monclient(hunting): authenticate will time out at 998.317750s 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.151+0000 7fd3bbfff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fd3bc13f650 0x7fd3bc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:52885/0 (socket says 172.21.15.151:52885) 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.151+0000 7fd3c122a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd3bc0a9d70 0x7fd3bc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:49149/0 (socket says 172.21.15.151:49149) 2024-03-29T10:37:47.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.151+0000 7fd3c122a640 1 -- 172.21.15.151:0/4106720185 learned_addr learned my addr 172.21.15.151:0/4106720185 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451159129 0 0) 0x7fd3bc145eb0 con 0x7fd3bc13f650 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a4003520 con 0x7fd3bc13f650 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2737004935 0 0) 0x7fd3a4003520 con 0x7fd3bc13f650 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 >> v1:172.21.15.169:6789/0 conn(0x7fd3bc1457f0 legacy=0x7fd3bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 >> v1:172.21.15.151:6789/0 conn(0x7fd3bc0a9d70 legacy=0x7fd3bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 monclient: found mon.noname-b 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:47.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc146340 con 0x7fd3bc13f650 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3c2c8d640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3c2c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd3b0001670 con 0x7fd3bc13f650 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:47.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3bb7fe640 10 monclient: dump: 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.154+0000 7fd3c2c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3b0001900 con 0x7fd3bc13f650 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 1 -- 172.21.15.151:0/4106720185 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd3b0001670 con 0x7fd3bc13f650 2024-03-29T10:37:47.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3bb7fe640 10 monclient: dump: 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3c2c8d640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:47.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.157+0000 7fd3c2c8d640 4 set_mon_vals no callback set 2024-03-29T10:37:47.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.158+0000 7fd3c2c8d640 10 monclient: shutdown 2024-03-29T10:37:47.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.158+0000 7fd3c2c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:47.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.158+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/4106720185 >> v1:172.21.15.151:6790/0 conn(0x7fd3bc13f650 legacy=0x7fd3bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.158+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/4106720185 shutdown_connections 2024-03-29T10:37:47.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.158+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/4106720185 >> 172.21.15.151:0/4106720185 conn(0x7fd3bc066a00 msgr2=0x7fd3bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.159+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/4106720185 shutdown_connections 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.159+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/4106720185 wait complete. 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.160+0000 7fd3c2c8d640 10 monclient: build_initial_monmap 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.160+0000 7fd3c2c8d640 10 monclient: monmap: 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:47.145 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.161+0000 7fd3c2c8d640 1 librados: starting msgr at 2024-03-29T10:37:47.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.161+0000 7fd3c2c8d640 1 librados: starting objecter 2024-03-29T10:37:47.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.162+0000 7fd3c2c8d640 1 Processor -- start 2024-03-29T10:37:47.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.162+0000 7fd3c2c8d640 1 -- start start 2024-03-29T10:37:47.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.162+0000 7fd3c2c8d640 1 librados: setting wanted keys 2024-03-29T10:37:47.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.162+0000 7fd3c2c8d640 1 librados: calling monclient init 2024-03-29T10:37:47.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.162+0000 7fd3c2c8d640 10 monclient: init 2024-03-29T10:37:47.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:47.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:47.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd3bc0a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:47.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd3bc13f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd3bc1457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc140620 con 0x7fd3bc1457f0 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc140ab0 con 0x7fd3bc13f650 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc141e90 con 0x7fd3bc0a9d70 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:47.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.163+0000 7fd3c2c8d640 10 monclient(hunting): authenticate will time out at 998.333432s 2024-03-29T10:37:47.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.168+0000 7fd3c122a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd3bc1457f0 0x7fd3bc1ddc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:49239/0 (socket says 172.21.15.151:49239) 2024-03-29T10:37:47.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.168+0000 7fd3c122a640 1 -- 172.21.15.151:0/611937278 learned_addr learned my addr 172.21.15.151:0/611937278 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:47.153 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.168+0000 7fd3b97fa640 10 client.?.objecter ms_handle_connect 0x7fd3bc1457f0 2024-03-29T10:37:47.153 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.168+0000 7fd3b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:47.153 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071027848 0 0) 0x7fd3bc140620 con 0x7fd3bc1457f0 2024-03-29T10:37:47.153 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:47.153 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:47.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:47.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:37:47.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a8004eb0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 client.?.objecter ms_handle_connect 0x7fd3bc0a9d70 2024-03-29T10:37:47.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.169+0000 7fd3b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1907955595 0 0) 0x7fd3a8004eb0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3bc140620 con 0x7fd3bc1457f0 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd3ac001830 con 0x7fd3bc1457f0 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 10 monclient(hunting): dump: 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:47.155 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.156 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:47.156 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:47.156 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:47.156 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.156 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.170+0000 7fd3b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:47.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195286686 0 0) 0x7fd3bc141e90 con 0x7fd3bc0a9d70 2024-03-29T10:37:47.157 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.173+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a80056b0 con 0x7fd3bc0a9d70 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116426844 0 0) 0x7fd3bc140620 con 0x7fd3bc1457f0 2024-03-29T10:37:47.158 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 >> v1:172.21.15.151:6790/0 conn(0x7fd3bc0a9d70 legacy=0x7fd3bc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 >> v1:172.21.15.169:6789/0 conn(0x7fd3bc13f650 legacy=0x7fd3bc143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 monclient: found mon.a 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc142170 con 0x7fd3bc1457f0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: _finish_auth 0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3ac0047e0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd3ac001830 con 0x7fd3bc1457f0 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3b97fa640 10 monclient: dump: 2024-03-29T10:37:47.159 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:47.160 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3baffd640 4 set_mon_vals no callback set 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 10 monclient: _renew_subs 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc143a30 con 0x7fd3bc1457f0 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:47.161 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 10 monclient: _renew_subs 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc140ea0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.174+0000 7fd3c2c8d640 1 librados: init done 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.175+0000 7fd3c2c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.176+0000 7fd3c2c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.176+0000 7fd3c2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.176+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc075f10 con 0x7fd3bc1457f0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.176+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fd3ac004a90 con 0x7fd3bc1457f0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fd3ac084710 con 0x7fd3bc1457f0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3b97fa640 10 client.4260.objecter ms_dispatch 0x7fd3bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3b97fa640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3b97fa640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:47.162 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3b97fa640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T10:37:47.163 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.177+0000 7fd3c0a29640 10 monclient: get_auth_request con 0x7fd3a806e240 auth_method 0 2024-03-29T10:37:47.165 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.181+0000 7fd3b97fa640 10 client.4260.objecter ms_handle_connect 0x7fd3a806e240 2024-03-29T10:37:47.169 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.184+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3ac04fff0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.169 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.184+0000 7fd3b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:47.169 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.184+0000 7fd3b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:47.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.514+0000 7fd3c2c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:47.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.514+0000 7fd3c2c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:47.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.514+0000 7fd3c2c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:37:47.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.514+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3bc0b3680 con 0x7fd3bc1457f0 2024-03-29T10:37:47.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.516+0000 7fd3b97fa640 1 -- 172.21.15.151:0/611937278 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fd3ac04fff0 con 0x7fd3bc1457f0 2024-03-29T10:37:47.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.516+0000 7fd3b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:47.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.516+0000 7fd3b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:47.501 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:47.501 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:47.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.519+0000 7fd3c2c8d640 10 librados: watch_flush enter 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.521+0000 7fd3c2c8d640 10 librados: watch_flush exit 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.521+0000 7fd3c2c8d640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.521+0000 7fd3c2c8d640 10 client.4260.objecter successfully canceled tick 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.521+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 >> v1:172.21.15.151:6816/1549873000 conn(0x7fd3a806e240 legacy=0x7fd3a8070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.522+0000 7fd3c2c8d640 10 monclient: shutdown 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.522+0000 7fd3c2c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:47.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.522+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 >> v1:172.21.15.151:6789/0 conn(0x7fd3bc1457f0 legacy=0x7fd3bc1ddc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:47.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.523+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 shutdown_connections 2024-03-29T10:37:47.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.523+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 >> 172.21.15.151:0/611937278 conn(0x7fd3bc066a00 msgr2=0x7fd3bc13cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:47.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.523+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 shutdown_connections 2024-03-29T10:37:47.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.523+0000 7fd3c2c8d640 1 -- 172.21.15.151:0/611937278 wait complete. 2024-03-29T10:37:47.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:47.523+0000 7fd3c2c8d640 1 librados: shutdown 2024-03-29T10:37:47.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:54.333 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:54.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.503+0000 7f6d8c9ec640 10 monclient: get_monmap_and_config 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.503+0000 7f6d8c9ec640 10 monclient: build_initial_monmap 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.503+0000 7f6d8c9ec640 10 monclient: monmap: 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:37:54.505495+0000 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:37:54.505495+0000 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:37:54.489 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.505+0000 7f6d8c9ec640 1 Processor -- start 2024-03-29T10:37:54.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.505+0000 7f6d8c9ec640 1 -- start start 2024-03-29T10:37:54.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.505+0000 7f6d8c9ec640 10 monclient: init 2024-03-29T10:37:54.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.506+0000 7f6d8c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:54.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.506+0000 7f6d8c9ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:54.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.509+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f6d80006a70 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.509+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f6d8002b100 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d877fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f6d80006a70 0x7f6d800c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:57869/0 (socket says 172.21.15.151:57869) 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d877fe640 1 -- 172.21.15.151:0/4073397355 learned_addr learned my addr 172.21.15.151:0/4073397355 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f6d80029c30 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8002a270 con 0x7f6d80029c30 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8002c440 con 0x7f6d80006a70 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800c9440 con 0x7f6d8002b100 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:54.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.510+0000 7f6d8c9ec640 10 monclient(hunting): authenticate will time out at 1005.681003s 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645058658 0 0) 0x7f6d8002c440 con 0x7f6d80006a70 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:37:54.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.513+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78003520 con 0x7f6d80006a70 2024-03-29T10:37:54.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2357083718 0 0) 0x7f6d78003520 con 0x7f6d80006a70 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 >> v1:172.21.15.151:6790/0 conn(0x7f6d8002b100 legacy=0x7f6d800cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 >> v1:172.21.15.151:6789/0 conn(0x7f6d80029c30 legacy=0x7f6d800d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 1 monclient: found mon.noname-c 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.514+0000 7f6d867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800c95d0 con 0x7f6d80006a70 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d7c001690 con 0x7f6d80006a70 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d8c9ec640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:37:54.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d8c9ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: dump: 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:54.502 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d7c0019a0 con 0x7f6d80006a70 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d8c9ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 1 -- 172.21.15.151:0/4073397355 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d7c001d30 con 0x7f6d80006a70 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d8c9ec640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d8c9ec640 4 set_mon_vals no callback set 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.515+0000 7f6d867fc640 10 monclient: dump: 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.503 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.516+0000 7f6d8c9ec640 10 monclient: shutdown 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.516+0000 7f6d8c9ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.516+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 >> v1:172.21.15.169:6789/0 conn(0x7f6d80006a70 legacy=0x7f6d800c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.518+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 shutdown_connections 2024-03-29T10:37:54.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.518+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 >> 172.21.15.151:0/4073397355 conn(0x7f6d800058f0 msgr2=0x7f6d80005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:54.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.521+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 shutdown_connections 2024-03-29T10:37:54.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.521+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/4073397355 wait complete. 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.523+0000 7f6d8c9ec640 10 monclient: build_initial_monmap 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.523+0000 7f6d8c9ec640 10 monclient: monmap: 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:37:54.508 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:37:54.509 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.523+0000 7f6d8c9ec640 1 librados: starting msgr at 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.523+0000 7f6d8c9ec640 1 librados: starting objecter 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.523+0000 7f6d8c9ec640 1 Processor -- start 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.524+0000 7f6d8c9ec640 1 -- start start 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.524+0000 7f6d8c9ec640 1 librados: setting wanted keys 2024-03-29T10:37:54.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.524+0000 7f6d8c9ec640 1 librados: calling monclient init 2024-03-29T10:37:54.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.524+0000 7f6d8c9ec640 10 monclient: init 2024-03-29T10:37:54.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.525+0000 7f6d8c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:54.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.525+0000 7f6d8c9ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:54.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.525+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f6d80006a70 addr v1:172.21.15.151:6789/0 2024-03-29T10:37:54.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.526+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f6d80029c30 addr v1:172.21.15.151:6790/0 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.530+0000 7f6d8c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f6d800320c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.530+0000 7f6d8c9ec640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800c2d20 con 0x7f6d80006a70 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.530+0000 7f6d8c9ec640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800c2eb0 con 0x7f6d800320c0 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.530+0000 7f6d8c9ec640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800c3040 con 0x7f6d80029c30 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.531+0000 7f6d8c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:54.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.531+0000 7f6d8c9ec640 10 monclient(hunting): authenticate will time out at 1005.701636s 2024-03-29T10:37:54.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.532+0000 7f6d86ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f6d80029c30 0x7f6d801646d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:51059/0 (socket says 172.21.15.151:51059) 2024-03-29T10:37:54.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.532+0000 7f6d86ffd640 1 -- 172.21.15.151:0/1784021738 learned_addr learned my addr 172.21.15.151:0/1784021738 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 client.?.objecter ms_handle_connect 0x7f6d80029c30 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 client.?.objecter ms_handle_connect 0x7f6d800320c0 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878565106 0 0) 0x7f6d800c3040 con 0x7f6d80029c30 2024-03-29T10:37:54.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:54.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:54.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:54.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:37:54.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.533+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c004eb0 con 0x7f6d80029c30 2024-03-29T10:37:54.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 client.?.objecter ms_handle_connect 0x7f6d80006a70 2024-03-29T10:37:54.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660179391 0 0) 0x7f6d800c2eb0 con 0x7f6d800320c0 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d800c3040 con 0x7f6d800320c0 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054462267 0 0) 0x7f6d6c004eb0 con 0x7f6d80029c30 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d800c2eb0 con 0x7f6d80029c30 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d74003f30 con 0x7f6d80029c30 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): dump: 2024-03-29T10:37:54.522 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.534+0000 7f6d73fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747930484 0 0) 0x7f6d800c3040 con 0x7f6d800320c0 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6c004eb0 con 0x7f6d800320c0 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382738577 0 0) 0x7f6d800c2d20 con 0x7f6d80006a70 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:54.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d800c3040 con 0x7f6d80006a70 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.535+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d8804fdd0 con 0x7f6d800320c0 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:54.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient(hunting): dump: 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3573160924 0 0) 0x7f6d6c004eb0 con 0x7f6d800320c0 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 >> v1:172.21.15.151:6790/0 conn(0x7f6d80029c30 legacy=0x7f6d801646d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 >> v1:172.21.15.151:6789/0 conn(0x7f6d80006a70 legacy=0x7f6d800c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 monclient: found mon.b 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d80164de0 con 0x7f6d800320c0 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient: _finish_auth 0 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:54.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 10 monclient: _renew_subs 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d80165030 con 0x7f6d800320c0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 10 monclient: _renew_subs 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8002e0c0 con 0x7f6d800320c0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d8c9ec640 1 librados: init done 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d8804f400 con 0x7f6d800320c0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.536+0000 7f6d73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.537+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d88075710 con 0x7f6d800320c0 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.537+0000 7f6d73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:54.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.537+0000 7f6d73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.537+0000 7f6d73fff640 10 monclient: dump: 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.538+0000 7f6d857fa640 4 set_mon_vals no callback set 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.539+0000 7f6d8c9ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.539+0000 7f6d8c9ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.539+0000 7f6d8c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.539+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68005080 con 0x7f6d800320c0 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.539+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f6d88067bf0 con 0x7f6d800320c0 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f6d880fe1b0 con 0x7f6d800320c0 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 10 client.4267.objecter ms_dispatch 0x7f6d88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d877fe640 10 monclient: get_auth_request con 0x7f6d6c06dd00 auth_method 0 2024-03-29T10:37:54.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.540+0000 7f6d73fff640 10 client.4267.objecter ms_handle_connect 0x7f6d6c06dd00 2024-03-29T10:37:54.535 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.548+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d880c9a90 con 0x7f6d800320c0 2024-03-29T10:37:54.535 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.548+0000 7f6d73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:54.535 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.548+0000 7f6d73fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:54.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.875+0000 7f6d8c9ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:54.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.875+0000 7f6d8c9ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:54.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.875+0000 7f6d8c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:37:54.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.875+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d68005330 con 0x7f6d800320c0 2024-03-29T10:37:54.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.879+0000 7f6d73fff640 1 -- 172.21.15.151:0/1784021738 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f6d880cb9c0 con 0x7f6d800320c0 2024-03-29T10:37:54.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.879+0000 7f6d73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:54.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.879+0000 7f6d73fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:54.865 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:37:54.865 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:54.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.883+0000 7f6d8c9ec640 10 librados: watch_flush enter 2024-03-29T10:37:54.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.883+0000 7f6d8c9ec640 10 librados: watch_flush exit 2024-03-29T10:37:54.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.883+0000 7f6d8c9ec640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T10:37:54.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.883+0000 7f6d8c9ec640 10 client.4267.objecter successfully canceled tick 2024-03-29T10:37:54.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 >> v1:172.21.15.151:6816/1549873000 conn(0x7f6d6c06dd00 legacy=0x7f6d6c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 10 monclient: shutdown 2024-03-29T10:37:54.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:54.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 >> v1:172.21.15.169:6789/0 conn(0x7f6d800320c0 legacy=0x7f6d8002d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:54.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 shutdown_connections 2024-03-29T10:37:54.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.884+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 >> 172.21.15.151:0/1784021738 conn(0x7f6d800058f0 msgr2=0x7f6d800c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:54.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.885+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 shutdown_connections 2024-03-29T10:37:54.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.886+0000 7f6d8c9ec640 1 -- 172.21.15.151:0/1784021738 wait complete. 2024-03-29T10:37:54.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:37:54.886+0000 7f6d8c9ec640 1 librados: shutdown 2024-03-29T10:37:54.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:01.695 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:01.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.877+0000 7ff6b283f640 10 monclient: get_monmap_and_config 2024-03-29T10:38:01.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.877+0000 7ff6b283f640 10 monclient: build_initial_monmap 2024-03-29T10:38:01.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.877+0000 7ff6b283f640 10 monclient: monmap: 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:01.877994+0000 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:01.877994+0000 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.878+0000 7ff6b283f640 1 Processor -- start 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.878+0000 7ff6b283f640 1 -- start start 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.878+0000 7ff6b283f640 10 monclient: init 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ac0acfa0 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ac072f10 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ac0ad380 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac0aa9f0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac0ac4c0 con 0x7ff6ac072f10 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac0ac7a0 con 0x7ff6ac0acfa0 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:01.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.879+0000 7ff6b283f640 10 monclient(hunting): authenticate will time out at 1013.049285s 2024-03-29T10:38:01.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.887+0000 7ff6b183d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7ff6ac0acfa0 0x7ff6ac141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:43243/0 (socket says 172.21.15.151:43243) 2024-03-29T10:38:01.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.887+0000 7ff6b183d640 1 -- 172.21.15.151:0/1476002941 learned_addr learned my addr 172.21.15.151:0/1476002941 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:01.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439985368 0 0) 0x7ff6ac0ac4c0 con 0x7ff6ac072f10 2024-03-29T10:38:01.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:01.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:01.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff698003520 con 0x7ff6ac072f10 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371579309 0 0) 0x7ff6ac0ac7a0 con 0x7ff6ac0acfa0 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:38:01.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.888+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ac0ac4c0 con 0x7ff6ac0acfa0 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742805569 0 0) 0x7ff698003520 con 0x7ff6ac072f10 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 >> v1:172.21.15.151:6790/0 conn(0x7ff6ac0acfa0 legacy=0x7ff6ac141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 >> v1:172.21.15.151:6789/0 conn(0x7ff6ac0ad380 legacy=0x7ff6ac1497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: _finish_hunting 0 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 monclient: found mon.noname-c 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:01.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac0ab5a0 con 0x7ff6ac072f10 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: _finish_auth 0 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: _check_auth_tickets 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff69c0015c0 con 0x7ff6ac072f10 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.891+0000 7ff6b083b640 10 monclient: dump: 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:01.877 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff69c001720 con 0x7ff6ac072f10 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 1 -- 172.21.15.151:0/1476002941 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff69c001b30 con 0x7ff6ac072f10 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.892+0000 7ff6b083b640 10 monclient: dump: 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:01.878 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.895+0000 7ff6b283f640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:38:01.879 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.895+0000 7ff6b283f640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.895+0000 7ff6b283f640 4 set_mon_vals no callback set 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.896+0000 7ff6b283f640 10 monclient: shutdown 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.896+0000 7ff6b283f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.896+0000 7ff6b283f640 1 -- 172.21.15.151:0/1476002941 >> v1:172.21.15.169:6789/0 conn(0x7ff6ac072f10 legacy=0x7ff6ac141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.896+0000 7ff6b283f640 1 -- 172.21.15.151:0/1476002941 shutdown_connections 2024-03-29T10:38:01.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.896+0000 7ff6b283f640 1 -- 172.21.15.151:0/1476002941 >> 172.21.15.151:0/1476002941 conn(0x7ff6ac0b3a30 msgr2=0x7ff6ac0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:01.886 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.900+0000 7ff6b283f640 1 -- 172.21.15.151:0/1476002941 shutdown_connections 2024-03-29T10:38:01.886 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.900+0000 7ff6b283f640 1 -- 172.21.15.151:0/1476002941 wait complete. 2024-03-29T10:38:01.886 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 10 monclient: build_initial_monmap 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 10 monclient: monmap: 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 librados: starting msgr at 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 librados: starting objecter 2024-03-29T10:38:01.887 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 Processor -- start 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 -- start start 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 librados: setting wanted keys 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 1 librados: calling monclient init 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.901+0000 7ff6b283f640 10 monclient: init 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ac072f10 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ac0acfa0 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:01.891 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ac0ad380 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac134b30 con 0x7ff6ac0ad380 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac13e640 con 0x7ff6ac0acfa0 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac1db980 con 0x7ff6ac072f10 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.902+0000 7ff6b283f640 10 monclient(hunting): authenticate will time out at 1013.072315s 2024-03-29T10:38:01.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.907+0000 7ff6b183d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7ff6ac072f10 0x7ff6ac1db270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:43463/0 (socket says 172.21.15.151:43463) 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.907+0000 7ff6b183d640 1 -- 172.21.15.151:0/1782552470 learned_addr learned my addr 172.21.15.151:0/1782552470 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 client.?.objecter ms_handle_connect 0x7ff6ac0ad380 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064406694 0 0) 0x7ff6ac134b30 con 0x7ff6ac0ad380 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:38:01.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.909+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff690004eb0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882919643 0 0) 0x7ff690004eb0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ac134b30 con 0x7ff6ac0ad380 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6a4001ad0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 10 monclient(hunting): dump: 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:01.894 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506194919 0 0) 0x7ff6ac134b30 con 0x7ff6ac0ad380 2024-03-29T10:38:01.895 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.910+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 >> v1:172.21.15.151:6790/0 conn(0x7ff6ac072f10 legacy=0x7ff6ac1db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.914+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 >> v1:172.21.15.169:6789/0 conn(0x7ff6ac0acfa0 legacy=0x7ff6ac133d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.914+0000 7ff6a27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.914+0000 7ff6a27fc640 1 monclient: found mon.a 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac13dda0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:01.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 client.?.objecter ms_handle_connect 0x7ff6ac0acfa0 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 client.?.objecter ms_handle_connect 0x7ff6ac072f10 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 10 monclient: _renew_subs 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac1ca770 con 0x7ff6ac0ad380 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 10 monclient: _renew_subs 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ac13ed40 con 0x7ff6ac0ad380 2024-03-29T10:38:01.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.915+0000 7ff6b283f640 1 librados: init done 2024-03-29T10:38:01.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.916+0000 7ff6b283f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:01.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.918+0000 7ff6b283f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:01.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.918+0000 7ff6b283f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:01.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.918+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ac1c7c40 con 0x7ff6ac0ad380 2024-03-29T10:38:01.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.921+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6a40020a0 con 0x7ff6ac0ad380 2024-03-29T10:38:01.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.921+0000 7ff6a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:01.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.921+0000 7ff6a37fe640 4 set_mon_vals no callback set 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6a4003b10 con 0x7ff6ac0ad380 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 10 monclient: dump: 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:01.910 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7ff6a4019b60 con 0x7ff6ac0ad380 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7ff6a4083c00 con 0x7ff6ac0ad380 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 10 client.4269.objecter ms_dispatch 0x7ff6ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.923+0000 7ff6a27fc640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.924+0000 7ff6a27fc640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T10:38:01.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.925+0000 7ff6b183d640 10 monclient: get_auth_request con 0x7ff69006dc70 auth_method 0 2024-03-29T10:38:01.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.925+0000 7ff6a27fc640 10 client.4269.objecter ms_handle_connect 0x7ff69006dc70 2024-03-29T10:38:01.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.931+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6a404f560 con 0x7ff6ac0ad380 2024-03-29T10:38:01.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.931+0000 7ff6a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:01.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:01.931+0000 7ff6a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:02.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.257+0000 7ff6b283f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:02.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.257+0000 7ff6b283f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:02.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.257+0000 7ff6b283f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:02.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.257+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6ac04f730 con 0x7ff6ac0ad380 2024-03-29T10:38:02.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.260+0000 7ff6a27fc640 1 -- 172.21.15.151:0/1782552470 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7ff6a4051490 con 0x7ff6ac0ad380 2024-03-29T10:38:02.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.260+0000 7ff6a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:02.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.260+0000 7ff6a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:02.246 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:02.247 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 10 librados: watch_flush enter 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 10 librados: watch_flush exit 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 10 client.4269.objecter successfully canceled tick 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 >> v1:172.21.15.151:6816/1549873000 conn(0x7ff69006dc70 legacy=0x7ff690070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.266+0000 7ff6b283f640 10 monclient: shutdown 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.267+0000 7ff6b283f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:02.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.267+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 >> v1:172.21.15.151:6789/0 conn(0x7ff6ac0ad380 legacy=0x7ff6ac134420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:02.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.268+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 shutdown_connections 2024-03-29T10:38:02.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.268+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 >> 172.21.15.151:0/1782552470 conn(0x7ff6ac0b3a30 msgr2=0x7ff6ac0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:02.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.269+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 shutdown_connections 2024-03-29T10:38:02.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.269+0000 7ff6b283f640 1 -- 172.21.15.151:0/1782552470 wait complete. 2024-03-29T10:38:02.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:02.269+0000 7ff6b283f640 1 librados: shutdown 2024-03-29T10:38:02.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:09.079 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.243+0000 7fa0fc9e8640 10 monclient: get_monmap_and_config 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.243+0000 7fa0fc9e8640 10 monclient: build_initial_monmap 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.243+0000 7fa0fc9e8640 10 monclient: monmap: 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:09.244847+0000 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:09.244847+0000 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:09.228 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.245+0000 7fa0fc9e8640 1 Processor -- start 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.245+0000 7fa0fc9e8640 1 -- start start 2024-03-29T10:38:09.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.245+0000 7fa0fc9e8640 10 monclient: init 2024-03-29T10:38:09.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f8074650 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f8070d50 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f813d520 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8072050 con 0x7fa0f8070d50 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8070890 con 0x7fa0f8074650 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80722a0 con 0x7fa0f813d520 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0fc9e8640 10 monclient(hunting): authenticate will time out at 1020.415710s 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0f7fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fa0f813d520 0x7fa0f814b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56942/0 (socket says 172.21.15.151:56942) 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.246+0000 7fa0f7fff640 1 -- 172.21.15.151:0/3224871134 learned_addr learned my addr 172.21.15.151:0/3224871134 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:09.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992387066 0 0) 0x7fa0f80722a0 con 0x7fa0f813d520 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e4003520 con 0x7fa0f813d520 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450261230 0 0) 0x7fa0f8070890 con 0x7fa0f8074650 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f80722a0 con 0x7fa0f8074650 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122148382 0 0) 0x7fa0f80722a0 con 0x7fa0f8074650 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 >> v1:172.21.15.151:6790/0 conn(0x7fa0f813d520 legacy=0x7fa0f814b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 >> v1:172.21.15.151:6789/0 conn(0x7fa0f8070d50 legacy=0x7fa0f813ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 monclient: found mon.noname-c 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f8072580 con 0x7fa0f8074650 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0e80019f0 con 0x7fa0f8074650 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.247+0000 7fa0f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: dump: 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.236 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0fc9e8640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0fc9e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8001b50 con 0x7fa0f8074650 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 1 -- 172.21.15.151:0/3224871134 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0e8001fa0 con 0x7fa0f8074650 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0f67fc640 10 monclient: dump: 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0fc9e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.248+0000 7fa0fc9e8640 4 set_mon_vals no callback set 2024-03-29T10:38:09.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.249+0000 7fa0fc9e8640 10 monclient: shutdown 2024-03-29T10:38:09.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.249+0000 7fa0fc9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:09.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.249+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/3224871134 >> v1:172.21.15.169:6789/0 conn(0x7fa0f8074650 legacy=0x7fa0f81431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.249+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/3224871134 shutdown_connections 2024-03-29T10:38:09.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.249+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/3224871134 >> 172.21.15.151:0/3224871134 conn(0x7fa0f80b4200 msgr2=0x7fa0f80b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:09.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.250+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/3224871134 shutdown_connections 2024-03-29T10:38:09.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.250+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/3224871134 wait complete. 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 10 monclient: build_initial_monmap 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 10 monclient: monmap: 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 librados: starting msgr at 2024-03-29T10:38:09.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 librados: starting objecter 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 Processor -- start 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 -- start start 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 librados: setting wanted keys 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 1 librados: calling monclient init 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.251+0000 7fa0fc9e8640 10 monclient: init 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f8070d50 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f8074650 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f813d520 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f813bd10 con 0x7fa0f813d520 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f811c8b0 con 0x7fa0f8070d50 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f811cb90 con 0x7fa0f8074650 2024-03-29T10:38:09.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.252+0000 7fa0fc9e8640 10 monclient(hunting): authenticate will time out at 1020.422372s 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0f77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa0f8070d50 0x7fa0f81de940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:52180/0 (socket says 172.21.15.151:52180) 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0f77fe640 1 -- 172.21.15.151:0/4244965734 learned_addr learned my addr 172.21.15.151:0/4244965734 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter ms_handle_connect 0x7fa0f8070d50 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter ms_handle_connect 0x7fa0f813d520 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter ms_handle_connect 0x7fa0f8074650 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145050625 0 0) 0x7fa0f811c8b0 con 0x7fa0f8070d50 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:09.251 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.253+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8004eb0 con 0x7fa0f8070d50 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537805694 0 0) 0x7fa0f813bd10 con 0x7fa0f813d520 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f811c8b0 con 0x7fa0f813d520 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368989294 0 0) 0x7fa0f811cb90 con 0x7fa0f8074650 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f813bd10 con 0x7fa0f8074650 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703535842 0 0) 0x7fa0f811c8b0 con 0x7fa0f813d520 2024-03-29T10:38:09.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f811cb90 con 0x7fa0f813d520 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0e0001850 con 0x7fa0f813d520 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): dump: 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.253 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373260430 0 0) 0x7fa0d8004eb0 con 0x7fa0f8070d50 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f811c8b0 con 0x7fa0f8070d50 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0e80019b0 con 0x7fa0f8070d50 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): dump: 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.254 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709918907 0 0) 0x7fa0f813bd10 con 0x7fa0f8074650 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0d8004eb0 con 0x7fa0f8074650 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0f00016b0 con 0x7fa0f8074650 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.254+0000 7fa0effff640 10 monclient(hunting): dump: 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.255 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2431611084 0 0) 0x7fa0f811cb90 con 0x7fa0f813d520 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 >> v1:172.21.15.151:6790/0 conn(0x7fa0f8074650 legacy=0x7fa0f80a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 >> v1:172.21.15.169:6789/0 conn(0x7fa0f8070d50 legacy=0x7fa0f81de940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 monclient: found mon.a 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f81df3c0 con 0x7fa0f813d520 2024-03-29T10:38:09.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: _finish_auth 0 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 5 monclient: authenticate success, global_id 4272 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 10 monclient: _renew_subs 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f80aa030 con 0x7fa0f813d520 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 10 monclient: _renew_subs 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f811fc80 con 0x7fa0f813d520 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0fc9e8640 1 librados: init done 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e0001c60 con 0x7fa0f813d520 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0f57fa640 4 set_mon_vals no callback set 2024-03-29T10:38:09.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa0e0004980 con 0x7fa0f813d520 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.255+0000 7fa0effff640 10 monclient: dump: 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:09.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fa0e001a950 con 0x7fa0f813d520 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0fc9e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0fc9e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0fc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc005080 con 0x7fa0f813d520 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fa0e0084950 con 0x7fa0f813d520 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 10 client.4272.objecter ms_dispatch 0x7fa0f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.259+0000 7fa0effff640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.260+0000 7fa0f77fe640 10 monclient: get_auth_request con 0x7fa0d806e090 auth_method 0 2024-03-29T10:38:09.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.260+0000 7fa0effff640 10 client.4272.objecter ms_handle_connect 0x7fa0d806e090 2024-03-29T10:38:09.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.271+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0e0050060 con 0x7fa0f813d520 2024-03-29T10:38:09.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.271+0000 7fa0effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:09.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.271+0000 7fa0effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:09.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.638+0000 7fa0fc9e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:09.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.638+0000 7fa0fc9e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:09.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.638+0000 7fa0fc9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:09.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.638+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0dc005330 con 0x7fa0f813d520 2024-03-29T10:38:09.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.640+0000 7fa0effff640 1 -- 172.21.15.151:0/4244965734 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fa0e0052160 con 0x7fa0f813d520 2024-03-29T10:38:09.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.640+0000 7fa0effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:09.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.640+0000 7fa0effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:09.624 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:09.624 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:09.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.644+0000 7fa0fc9e8640 10 librados: watch_flush enter 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 10 librados: watch_flush exit 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 10 client.4272.objecter successfully canceled tick 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 >> v1:172.21.15.151:6816/1549873000 conn(0x7fa0d806e090 legacy=0x7fa0d8070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 10 monclient: shutdown 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:09.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 >> v1:172.21.15.151:6789/0 conn(0x7fa0f813d520 legacy=0x7fa0f811f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:09.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 shutdown_connections 2024-03-29T10:38:09.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 >> 172.21.15.151:0/4244965734 conn(0x7fa0f80b4200 msgr2=0x7fa0f80b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:09.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 shutdown_connections 2024-03-29T10:38:09.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 -- 172.21.15.151:0/4244965734 wait complete. 2024-03-29T10:38:09.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:09.645+0000 7fa0fc9e8640 1 librados: shutdown 2024-03-29T10:38:09.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:16.453 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.601+0000 7fd0477db640 10 monclient: get_monmap_and_config 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.601+0000 7fd0477db640 10 monclient: build_initial_monmap 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.601+0000 7fd0477db640 10 monclient: monmap: 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:16.602862+0000 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:16.602862+0000 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:16.586 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.602+0000 7fd0477db640 1 Processor -- start 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.603+0000 7fd0477db640 1 -- start start 2024-03-29T10:38:16.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.603+0000 7fd0477db640 10 monclient: init 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.603+0000 7fd0477db640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.603+0000 7fd0477db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.603+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-b con 0x7fd0401457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-c con 0x7fd04013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-a con 0x7fd0400a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0400aaef0 con 0x7fd0400a9d70 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd040145bd0 con 0x7fd04013f650 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd040145eb0 con 0x7fd0401457f0 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:16.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd0477db640 10 monclient(hunting): authenticate will time out at 1027.773593s 2024-03-29T10:38:16.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd045577640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fd0401457f0 0x7fd04013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:44938/0 (socket says 172.21.15.151:44938) 2024-03-29T10:38:16.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.604+0000 7fd045577640 1 -- 172.21.15.151:0/2662304491 learned_addr learned my addr 172.21.15.151:0/2662304491 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:16.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126722733 0 0) 0x7fd040145eb0 con 0x7fd0401457f0 2024-03-29T10:38:16.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:16.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02c003520 con 0x7fd0401457f0 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339227010 0 0) 0x7fd040145bd0 con 0x7fd04013f650 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd040145eb0 con 0x7fd04013f650 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031915356 0 0) 0x7fd040145eb0 con 0x7fd04013f650 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 >> v1:172.21.15.151:6790/0 conn(0x7fd0401457f0 legacy=0x7fd04013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:16.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.605+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 >> v1:172.21.15.151:6789/0 conn(0x7fd0400a9d70 legacy=0x7fd04014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 1 monclient: found mon.noname-c 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd040146340 con 0x7fd04013f650 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd030001520 con 0x7fd04013f650 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd037fff640 10 monclient: dump: 2024-03-29T10:38:16.591 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd0477db640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:38:16.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.607+0000 7fd0477db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd030001680 con 0x7fd04013f650 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd0477db640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd0477db640 4 set_mon_vals no callback set 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 1 -- 172.21.15.151:0/2662304491 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd030001ad0 con 0x7fd04013f650 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.609+0000 7fd037fff640 10 monclient: dump: 2024-03-29T10:38:16.593 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:16.594 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 10 monclient: shutdown 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 1 -- 172.21.15.151:0/2662304491 >> v1:172.21.15.169:6789/0 conn(0x7fd04013f650 legacy=0x7fd0400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 1 -- 172.21.15.151:0/2662304491 shutdown_connections 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 1 -- 172.21.15.151:0/2662304491 >> 172.21.15.151:0/2662304491 conn(0x7fd040066a00 msgr2=0x7fd040066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 1 -- 172.21.15.151:0/2662304491 shutdown_connections 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.610+0000 7fd0477db640 1 -- 172.21.15.151:0/2662304491 wait complete. 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.611+0000 7fd0477db640 10 monclient: build_initial_monmap 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.611+0000 7fd0477db640 10 monclient: monmap: 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:16.596 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.611+0000 7fd0477db640 1 librados: starting msgr at 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.611+0000 7fd0477db640 1 librados: starting objecter 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 Processor -- start 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 -- start start 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 librados: setting wanted keys 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 librados: calling monclient init 2024-03-29T10:38:16.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient: init 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-a con 0x7fd0400a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-b con 0x7fd04013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 10 monclient(hunting): picked mon.noname-c con 0x7fd0401457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0401359f0 con 0x7fd0400a9d70 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd0477db640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0401419c0 con 0x7fd04013f650 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd045d78640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fd0401457f0 0x7fd0401da9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45884/0 (socket says 172.21.15.151:45884) 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.612+0000 7fd045d78640 1 -- 172.21.15.151:0/3084909597 learned_addr learned my addr 172.21.15.151:0/3084909597 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:16.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd040141ef0 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd0477db640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 client.?.objecter ms_handle_connect 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd0477db640 10 monclient(hunting): authenticate will time out at 1027.782687s 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126656021 0 0) 0x7fd040141ef0 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024004eb0 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 703113658 0 0) 0x7fd024004eb0 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd040141ef0 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd038001c60 con 0x7fd0401457f0 2024-03-29T10:38:16.601 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.613+0000 7fd035ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968781611 0 0) 0x7fd040141ef0 con 0x7fd0401457f0 2024-03-29T10:38:16.602 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 >> v1:172.21.15.169:6789/0 conn(0x7fd04013f650 legacy=0x7fd040142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 >> v1:172.21.15.151:6789/0 conn(0x7fd0400a9d70 legacy=0x7fd040140280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 1 monclient: found mon.c 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.614+0000 7fd035ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04013b7a0 con 0x7fd0401457f0 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 10 client.?.objecter ms_handle_connect 0x7fd04013f650 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd035ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:38:16.603 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 10 monclient: _renew_subs 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0401da440 con 0x7fd0401457f0 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 10 monclient: _renew_subs 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd040142d00 con 0x7fd0401457f0 2024-03-29T10:38:16.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.615+0000 7fd0477db640 1 librados: init done 2024-03-29T10:38:16.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd038003970 con 0x7fd0401457f0 2024-03-29T10:38:16.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd038003d80 con 0x7fd0401457f0 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 10 monclient: dump: 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:16.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd0377fe640 4 set_mon_vals no callback set 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.617+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fd038003fa0 con 0x7fd0401457f0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fd038084d90 con 0x7fd0401457f0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd035ffb640 10 client.4268.objecter ms_dispatch 0x7fd040000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd035ffb640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd035ffb640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd035ffb640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd0477db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd0477db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd0477db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd020005080 con 0x7fd0401457f0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.618+0000 7fd045577640 10 monclient: get_auth_request con 0x7fd02406db10 auth_method 0 2024-03-29T10:38:16.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.619+0000 7fd035ffb640 10 client.4268.objecter ms_handle_connect 0x7fd02406db10 2024-03-29T10:38:16.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.626+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd038050850 con 0x7fd0401457f0 2024-03-29T10:38:16.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.626+0000 7fd035ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:16.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.626+0000 7fd035ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:16.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.938+0000 7fd0477db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:16.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.938+0000 7fd0477db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:16.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.938+0000 7fd0477db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:16.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.938+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd020005330 con 0x7fd0401457f0 2024-03-29T10:38:16.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.939+0000 7fd035ffb640 1 -- 172.21.15.151:0/3084909597 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fd038084440 con 0x7fd0401457f0 2024-03-29T10:38:16.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.939+0000 7fd035ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:16.924 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:16.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.939+0000 7fd035ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:16.924 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:16.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.943+0000 7fd0477db640 10 librados: watch_flush enter 2024-03-29T10:38:16.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.943+0000 7fd0477db640 10 librados: watch_flush exit 2024-03-29T10:38:16.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.943+0000 7fd0477db640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T10:38:16.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.943+0000 7fd0477db640 10 client.4268.objecter successfully canceled tick 2024-03-29T10:38:16.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 >> v1:172.21.15.151:6816/1549873000 conn(0x7fd02406db10 legacy=0x7fd02406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:16.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 10 monclient: shutdown 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 >> v1:172.21.15.151:6790/0 conn(0x7fd0401457f0 legacy=0x7fd0401da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 shutdown_connections 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 >> 172.21.15.151:0/3084909597 conn(0x7fd040066a00 msgr2=0x7fd0400b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 shutdown_connections 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 -- 172.21.15.151:0/3084909597 wait complete. 2024-03-29T10:38:16.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:16.944+0000 7fd0477db640 1 librados: shutdown 2024-03-29T10:38:16.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:23.749 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:23.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.913+0000 7f1d806ff640 10 monclient: get_monmap_and_config 2024-03-29T10:38:23.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.913+0000 7f1d806ff640 10 monclient: build_initial_monmap 2024-03-29T10:38:23.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.913+0000 7f1d806ff640 10 monclient: monmap: 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:23.914171+0000 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:23.914171+0000 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:23.903 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.904 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:23.904 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:23.904 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.915+0000 7f1d806ff640 1 Processor -- start 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.915+0000 7f1d806ff640 1 -- start start 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.915+0000 7f1d806ff640 10 monclient: init 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:23.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-b con 0x7f1d780ab360 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-a con 0x7f1d7814fb00 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-c con 0x7f1d78153090 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d780a94e0 con 0x7f1d7814fb00 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d780afb60 con 0x7f1d78153090 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d780aab80 con 0x7f1d780ab360 2024-03-29T10:38:23.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:23.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.916+0000 7f1d806ff640 10 monclient(hunting): authenticate will time out at 1035.086113s 2024-03-29T10:38:23.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7ec9c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1d78153090 0x7f1d7815cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:39366/0 (socket says 172.21.15.151:39366) 2024-03-29T10:38:23.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7ec9c640 1 -- 172.21.15.151:0/273966427 learned_addr learned my addr 172.21.15.151:0/273966427 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:23.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960061393 0 0) 0x7f1d780afb60 con 0x7f1d78153090 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 10 monclient(hunting): _negotiate 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.920+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d64003520 con 0x7f1d78153090 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418121561 0 0) 0x7f1d780a94e0 con 0x7f1d7814fb00 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 10 monclient(hunting): _negotiate 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d780afb60 con 0x7f1d7814fb00 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598763357 0 0) 0x7f1d64003520 con 0x7f1d78153090 2024-03-29T10:38:23.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.921+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 >> v1:172.21.15.151:6790/0 conn(0x7f1d780ab360 legacy=0x7f1d780ad450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 >> v1:172.21.15.151:6789/0 conn(0x7f1d7814fb00 legacy=0x7f1d78152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: _finish_hunting 0 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 monclient: found mon.noname-c 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d780aa680 con 0x7f1d78153090 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: _finish_auth 0 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: _check_auth_tickets 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d68001690 con 0x7f1d78153090 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: dump: 2024-03-29T10:38:23.917 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d806ff640 5 monclient: authenticate success, global_id 4285 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d806ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d680017f0 con 0x7f1d78153090 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:23.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 1 -- 172.21.15.151:0/273966427 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d68001c40 con 0x7f1d78153090 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d7d499640 10 monclient: dump: 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d806ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:23.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.923+0000 7f1d806ff640 4 set_mon_vals no callback set 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.924+0000 7f1d806ff640 10 monclient: shutdown 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.924+0000 7f1d806ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.924+0000 7f1d806ff640 1 -- 172.21.15.151:0/273966427 >> v1:172.21.15.169:6789/0 conn(0x7f1d78153090 legacy=0x7f1d7815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.925+0000 7f1d806ff640 1 -- 172.21.15.151:0/273966427 shutdown_connections 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.925+0000 7f1d806ff640 1 -- 172.21.15.151:0/273966427 >> 172.21.15.151:0/273966427 conn(0x7f1d780b35e0 msgr2=0x7f1d780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.925+0000 7f1d806ff640 1 -- 172.21.15.151:0/273966427 shutdown_connections 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.925+0000 7f1d806ff640 1 -- 172.21.15.151:0/273966427 wait complete. 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 10 monclient: build_initial_monmap 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 10 monclient: monmap: 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:23.920 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 librados: starting msgr at 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 librados: starting objecter 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 Processor -- start 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 -- start start 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 librados: setting wanted keys 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 1 librados: calling monclient init 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.926+0000 7f1d806ff640 10 monclient: init 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.927+0000 7f1d806ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.927+0000 7f1d806ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.927+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-a con 0x7f1d780ab360 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:23.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-c con 0x7f1d7814fb00 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 10 monclient(hunting): picked mon.noname-b con 0x7f1d78153090 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7814eea0 con 0x7f1d780ab360 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d78151c20 con 0x7f1d78153090 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d78151db0 con 0x7f1d7814fb00 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d806ff640 10 monclient(hunting): authenticate will time out at 1035.098042s 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d7ec9c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1d78153090 0x7f1d78150a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:40152/0 (socket says 172.21.15.151:40152) 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d7ec9c640 1 -- 172.21.15.151:0/3568189246 learned_addr learned my addr 172.21.15.151:0/3568189246 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d6effd640 10 client.?.objecter ms_handle_connect 0x7f1d78153090 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.928+0000 7f1d6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228111562 0 0) 0x7f1d78151c20 con 0x7f1d78153090 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:23.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:23.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:38:23.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.929+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c004eb0 con 0x7f1d78153090 2024-03-29T10:38:23.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563747697 0 0) 0x7f1d5c004eb0 con 0x7f1d78153090 2024-03-29T10:38:23.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d78151c20 con 0x7f1d78153090 2024-03-29T10:38:23.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d68001bc0 con 0x7f1d78153090 2024-03-29T10:38:23.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 10 monclient(hunting): dump: 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.930+0000 7f1d6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:23.928 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 client.?.objecter ms_handle_connect 0x7f1d7814fb00 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977212962 0 0) 0x7f1d78151db0 con 0x7f1d7814fb00 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c004eb0 con 0x7f1d7814fb00 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2571450121 0 0) 0x7f1d78151c20 con 0x7f1d78153090 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 >> v1:172.21.15.151:6790/0 conn(0x7f1d7814fb00 legacy=0x7f1d78150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 >> v1:172.21.15.151:6789/0 conn(0x7f1d780ab360 legacy=0x7f1d78151510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 monclient: found mon.b 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:23.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7806ea10 con 0x7f1d78153090 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: _finish_auth 0 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d68004a90 con 0x7f1d78153090 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1d68004da0 con 0x7f1d78153090 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d6effd640 10 monclient: dump: 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:23.930 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.933+0000 7f1d7cc98640 4 set_mon_vals no callback set 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.934+0000 7f1d806ff640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.934+0000 7f1d806ff640 10 monclient: _renew_subs 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7806ec60 con 0x7f1d78153090 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:23.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 10 monclient: _renew_subs 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d781f8590 con 0x7f1d78153090 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.935+0000 7f1d806ff640 1 librados: init done 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.937+0000 7f1d806ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.937+0000 7f1d806ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.937+0000 7f1d806ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.937+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7808bc00 con 0x7f1d78153090 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.938+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f1d6801ad70 con 0x7f1d78153090 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f1d68084af0 con 0x7f1d78153090 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 10 client.4288.objecter ms_dispatch 0x7f1d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d7e49b640 10 monclient: get_auth_request con 0x7f1d5c06dc90 auth_method 0 2024-03-29T10:38:23.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.939+0000 7f1d6effd640 10 client.4288.objecter ms_handle_connect 0x7f1d5c06dc90 2024-03-29T10:38:23.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.946+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d680503d0 con 0x7f1d78153090 2024-03-29T10:38:23.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.946+0000 7f1d6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:23.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:23.946+0000 7f1d6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:24.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.281+0000 7f1d806ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:24.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.281+0000 7f1d806ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:24.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.281+0000 7f1d806ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:24.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.281+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d78152260 con 0x7f1d78153090 2024-03-29T10:38:24.268 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:24.268 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:24.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.284+0000 7f1d6effd640 1 -- 172.21.15.151:0/3568189246 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f1d78152260 con 0x7f1d78153090 2024-03-29T10:38:24.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.284+0000 7f1d6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:24.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.284+0000 7f1d6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 10 librados: watch_flush enter 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 10 librados: watch_flush exit 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 >> v1:172.21.15.151:6816/1549873000 conn(0x7f1d5c06dc90 legacy=0x7f1d5c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 10 monclient: shutdown 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:24.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 >> v1:172.21.15.169:6789/0 conn(0x7f1d78153090 legacy=0x7f1d78150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:24.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 shutdown_connections 2024-03-29T10:38:24.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.287+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 >> 172.21.15.151:0/3568189246 conn(0x7f1d780b35e0 msgr2=0x7f1d7808a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:24.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.288+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 shutdown_connections 2024-03-29T10:38:24.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.288+0000 7f1d806ff640 1 -- 172.21.15.151:0/3568189246 wait complete. 2024-03-29T10:38:24.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:24.288+0000 7f1d806ff640 1 librados: shutdown 2024-03-29T10:38:24.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:31.096 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.267+0000 7f7878b53640 10 monclient: get_monmap_and_config 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.267+0000 7f7878b53640 10 monclient: build_initial_monmap 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.268+0000 7f7878b53640 10 monclient: monmap: 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:31.268935+0000 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:31.268935+0000 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:31.252 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:31.253 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:31.253 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.272+0000 7f7878b53640 1 Processor -- start 2024-03-29T10:38:31.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.272+0000 7f7878b53640 1 -- start start 2024-03-29T10:38:31.256 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.272+0000 7f7878b53640 10 monclient: init 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-b con 0x7f7874149a40 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-a con 0x7f7874071f80 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-c con 0x7f78740a9e00 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:31.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78740aa2b0 con 0x7f7874071f80 2024-03-29T10:38:31.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78740ab7a0 con 0x7f78740a9e00 2024-03-29T10:38:31.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78740aba80 con 0x7f7874149a40 2024-03-29T10:38:31.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:31.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.273+0000 7f7878b53640 10 monclient(hunting): authenticate will time out at 1042.443023s 2024-03-29T10:38:31.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.276+0000 7f7872ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7874071f80 0x7f78740a96f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:51815/0 (socket says 172.21.15.151:51815) 2024-03-29T10:38:31.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.276+0000 7f7872ffd640 1 -- 172.21.15.151:0/200981999 learned_addr learned my addr 172.21.15.151:0/200981999 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.277+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097846668 0 0) 0x7f78740aa2b0 con 0x7f7874071f80 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.277+0000 7f78727fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.277+0000 7f78727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:38:31.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c003520 con 0x7f7874071f80 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2945618701 0 0) 0x7f785c003520 con 0x7f7874071f80 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 >> v1:172.21.15.151:6790/0 conn(0x7f7874149a40 legacy=0x7f78740ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 >> v1:172.21.15.169:6789/0 conn(0x7f78740a9e00 legacy=0x7f7874151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.278+0000 7f78727fc640 1 monclient: found mon.noname-a 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787414a240 con 0x7f7874071f80 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f786c001a10 con 0x7f7874071f80 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: dump: 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.266 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f786c001ca0 con 0x7f7874071f80 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 1 -- 172.21.15.151:0/200981999 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f786c002030 con 0x7f7874071f80 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.279+0000 7f78727fc640 10 monclient: dump: 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.267 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.280+0000 7f7878b53640 5 monclient: authenticate success, global_id 4278 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.280+0000 7f7878b53640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.280+0000 7f7878b53640 4 set_mon_vals no callback set 2024-03-29T10:38:31.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.284+0000 7f7878b53640 10 monclient: shutdown 2024-03-29T10:38:31.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.284+0000 7f7878b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:31.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.284+0000 7f7878b53640 1 -- 172.21.15.151:0/200981999 >> v1:172.21.15.151:6789/0 conn(0x7f7874071f80 legacy=0x7f78740a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.287+0000 7f7878b53640 1 -- 172.21.15.151:0/200981999 shutdown_connections 2024-03-29T10:38:31.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.287+0000 7f7878b53640 1 -- 172.21.15.151:0/200981999 >> 172.21.15.151:0/200981999 conn(0x7f78740b3ae0 msgr2=0x7f78740b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:31.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.288+0000 7f7878b53640 1 -- 172.21.15.151:0/200981999 shutdown_connections 2024-03-29T10:38:31.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.288+0000 7f7878b53640 1 -- 172.21.15.151:0/200981999 wait complete. 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.291+0000 7f7878b53640 10 monclient: build_initial_monmap 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.291+0000 7f7878b53640 10 monclient: monmap: 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.292+0000 7f7878b53640 1 librados: starting msgr at 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.292+0000 7f7878b53640 1 librados: starting objecter 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.292+0000 7f7878b53640 1 Processor -- start 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.292+0000 7f7878b53640 1 -- start start 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.292+0000 7f7878b53640 1 librados: setting wanted keys 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.293+0000 7f7878b53640 1 librados: calling monclient init 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.293+0000 7f7878b53640 10 monclient: init 2024-03-29T10:38:31.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.293+0000 7f7878b53640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:31.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.294+0000 7f7878b53640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:31.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.297+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-a con 0x7f7874071f80 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:31.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.297+0000 7f78737fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7874071f80 0x7f78741dbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:52027/0 (socket says 172.21.15.151:52027) 2024-03-29T10:38:31.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-c con 0x7f78740a9e00 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 10 monclient(hunting): picked mon.noname-b con 0x7f7874149a40 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78741db650 con 0x7f7874071f80 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78741dc4d0 con 0x7f7874149a40 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78741414b0 con 0x7f78740a9e00 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.299+0000 7f7878b53640 10 monclient(hunting): authenticate will time out at 1042.468694s 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f7872ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f78740a9e00 0x7f78741d80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:47405/0 (socket says 172.21.15.151:47405) 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f7872ffd640 1 -- 172.21.15.151:0/2859954808 learned_addr learned my addr 172.21.15.151:0/2859954808 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 10 client.?.objecter ms_handle_connect 0x7f7874071f80 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 10 client.?.objecter ms_handle_connect 0x7f7874149a40 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:31.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258557209 0 0) 0x7f78741db650 con 0x7f7874071f80 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.304+0000 7f785bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7860004eb0 con 0x7f7874071f80 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 client.?.objecter ms_handle_connect 0x7f78740a9e00 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382694312 0 0) 0x7f78741dc4d0 con 0x7f7874149a40 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78741db650 con 0x7f7874149a40 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656940998 0 0) 0x7f78741414b0 con 0x7f78740a9e00 2024-03-29T10:38:31.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78741dc4d0 con 0x7f78740a9e00 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3534395432 0 0) 0x7f7860004eb0 con 0x7f7874071f80 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78741414b0 con 0x7f7874071f80 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f78640018d0 con 0x7f7874071f80 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.305+0000 7f785bfff640 10 monclient(hunting): dump: 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.297 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 219974009 0 0) 0x7f78741db650 con 0x7f7874149a40 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7860004eb0 con 0x7f7874149a40 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181623130 0 0) 0x7f78741dc4d0 con 0x7f78740a9e00 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78741db650 con 0x7f78740a9e00 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7868001710 con 0x7f7874149a40 2024-03-29T10:38:31.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): dump: 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f786c0018d0 con 0x7f78740a9e00 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient(hunting): dump: 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 333600678 0 0) 0x7f78741414b0 con 0x7f7874071f80 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 >> v1:172.21.15.151:6790/0 conn(0x7f78740a9e00 legacy=0x7f78741d80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 >> v1:172.21.15.169:6789/0 conn(0x7f7874149a40 legacy=0x7f78741d87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 monclient: found mon.a 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78741db8a0 con 0x7f7874071f80 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient: _finish_auth 0 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.306+0000 7f785bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 10 monclient: _renew_subs 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78741400e0 con 0x7f7874071f80 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 10 monclient: _renew_subs 2024-03-29T10:38:31.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7874071cf0 con 0x7f7874071f80 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.307+0000 7f7878b53640 1 librados: init done 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7864003610 con 0x7f7874071f80 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f78640039e0 con 0x7f7874071f80 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.309+0000 7f785bfff640 10 monclient: dump: 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:31.302 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:31.303 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:31.303 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:31.303 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:31.303 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.310+0000 7f7871ffb640 4 set_mon_vals no callback set 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.311+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f7864003b40 con 0x7f7874071f80 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.311+0000 7f7859ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.311+0000 7f7859ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.311+0000 7f7859ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.311+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7874074c60 con 0x7f7874071f80 2024-03-29T10:38:31.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.312+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f7864084950 con 0x7f7874071f80 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.312+0000 7f785bfff640 10 client.4281.objecter ms_dispatch 0x7f7874000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.312+0000 7f785bfff640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.312+0000 7f785bfff640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.312+0000 7f785bfff640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.313+0000 7f7872ffd640 10 monclient: get_auth_request con 0x7f786006e160 auth_method 0 2024-03-29T10:38:31.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.316+0000 7f785bfff640 10 client.4281.objecter ms_handle_connect 0x7f786006e160 2024-03-29T10:38:31.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.324+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7864050290 con 0x7f7874071f80 2024-03-29T10:38:31.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.324+0000 7f785bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:31.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.324+0000 7f785bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:31.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.649+0000 7f7859ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:31.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.649+0000 7f7859ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:31.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.649+0000 7f7859ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:31.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.649+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f787404f730 con 0x7f7874071f80 2024-03-29T10:38:31.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.651+0000 7f785bfff640 1 -- 172.21.15.151:0/2859954808 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f7864084000 con 0x7f7874071f80 2024-03-29T10:38:31.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.651+0000 7f785bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:31.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.651+0000 7f785bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:31.636 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:31.636 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:31.639 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.655+0000 7f7859ffb640 10 librados: watch_flush enter 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.655+0000 7f7859ffb640 10 librados: watch_flush exit 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.655+0000 7f7859ffb640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.655+0000 7f7859ffb640 10 client.4281.objecter successfully canceled tick 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.656+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 >> v1:172.21.15.151:6816/1549873000 conn(0x7f786006e160 legacy=0x7f7860070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.656+0000 7f7859ffb640 10 monclient: shutdown 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.656+0000 7f7859ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:31.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.656+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 >> v1:172.21.15.151:6789/0 conn(0x7f7874071f80 legacy=0x7f78741dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:31.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.659+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 shutdown_connections 2024-03-29T10:38:31.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.659+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 >> 172.21.15.151:0/2859954808 conn(0x7f78740b3ae0 msgr2=0x7f78740b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:31.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.659+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 shutdown_connections 2024-03-29T10:38:31.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.660+0000 7f7859ffb640 1 -- 172.21.15.151:0/2859954808 wait complete. 2024-03-29T10:38:31.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:31.660+0000 7f7859ffb640 1 librados: shutdown 2024-03-29T10:38:31.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:38.468 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.633+0000 7fcf409a7640 10 monclient: get_monmap_and_config 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.633+0000 7fcf409a7640 10 monclient: build_initial_monmap 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.633+0000 7fcf409a7640 10 monclient: monmap: 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:38.634426+0000 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:38.634426+0000 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:38.619 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.634+0000 7fcf409a7640 1 Processor -- start 2024-03-29T10:38:38.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.634+0000 7fcf409a7640 1 -- start start 2024-03-29T10:38:38.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.634+0000 7fcf409a7640 10 monclient: init 2024-03-29T10:38:38.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.635+0000 7fcf409a7640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:38.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.635+0000 7fcf409a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:38.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.638+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c0acfa0 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c072f10 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c0ad380 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0aa9f0 con 0x7fcf3c072f10 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ac4c0 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0ac7a0 con 0x7fcf3c0ad380 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:38.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.639+0000 7fcf409a7640 10 monclient(hunting): authenticate will time out at 1049.808874s 2024-03-29T10:38:38.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.642+0000 7fcf3ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcf3c0acfa0 0x7fcf3c141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:59531/0 (socket says 172.21.15.151:59531) 2024-03-29T10:38:38.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.642+0000 7fcf3ad9d640 1 -- 172.21.15.151:0/1875632357 learned_addr learned my addr 172.21.15.151:0/1875632357 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448066015 0 0) 0x7fcf3c0ac4c0 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf24003520 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985016110 0 0) 0x7fcf3c0aa9f0 con 0x7fcf3c072f10 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.643+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c0ac4c0 con 0x7fcf3c072f10 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591618690 0 0) 0x7fcf3c0ac7a0 con 0x7fcf3c0ad380 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c0aa9f0 con 0x7fcf3c0ad380 2024-03-29T10:38:38.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 383334219 0 0) 0x7fcf3c0ac4c0 con 0x7fcf3c072f10 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 >> v1:172.21.15.151:6790/0 conn(0x7fcf3c0ad380 legacy=0x7fcf3c1497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 >> v1:172.21.15.169:6789/0 conn(0x7fcf3c0acfa0 legacy=0x7fcf3c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 monclient: found mon.noname-a 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0ab5a0 con 0x7fcf3c072f10 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: _finish_auth 0 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcf2c0015f0 con 0x7fcf3c072f10 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:38.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: dump: 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf409a7640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:38:38.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf409a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf2c001750 con 0x7fcf3c072f10 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 1 -- 172.21.15.151:0/1875632357 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcf2c001ba0 con 0x7fcf3c072f10 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf39d9b640 10 monclient: dump: 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:38.632 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf409a7640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.645+0000 7fcf409a7640 4 set_mon_vals no callback set 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.646+0000 7fcf409a7640 10 monclient: shutdown 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.646+0000 7fcf409a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.646+0000 7fcf409a7640 1 -- 172.21.15.151:0/1875632357 >> v1:172.21.15.151:6789/0 conn(0x7fcf3c072f10 legacy=0x7fcf3c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.646+0000 7fcf409a7640 1 -- 172.21.15.151:0/1875632357 shutdown_connections 2024-03-29T10:38:38.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.646+0000 7fcf409a7640 1 -- 172.21.15.151:0/1875632357 >> 172.21.15.151:0/1875632357 conn(0x7fcf3c0b3a30 msgr2=0x7fcf3c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.647+0000 7fcf409a7640 1 -- 172.21.15.151:0/1875632357 shutdown_connections 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.648+0000 7fcf409a7640 1 -- 172.21.15.151:0/1875632357 wait complete. 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.649+0000 7fcf409a7640 10 monclient: build_initial_monmap 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.649+0000 7fcf409a7640 10 monclient: monmap: 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:38.634 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.649+0000 7fcf409a7640 1 librados: starting msgr at 2024-03-29T10:38:38.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.649+0000 7fcf409a7640 1 librados: starting objecter 2024-03-29T10:38:38.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.649+0000 7fcf409a7640 1 Processor -- start 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 1 -- start start 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 1 librados: setting wanted keys 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 1 librados: calling monclient init 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 10 monclient: init 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:38.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.653+0000 7fcf409a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c072f10 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c0acfa0 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c0ad380 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c070320 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c070a20 con 0x7fcf3c072f10 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c070bb0 con 0x7fcf3c0ad380 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:38.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.654+0000 7fcf409a7640 10 monclient(hunting): authenticate will time out at 1049.824224s 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf3ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcf3c072f10 0x7fcf3c139140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:59633/0 (socket says 172.21.15.151:59633) 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf3ad9d640 1 -- 172.21.15.151:0/1973975447 learned_addr learned my addr 172.21.15.151:0/1973975447 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 client.?.objecter ms_handle_connect 0x7fcf3c072f10 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 client.?.objecter ms_handle_connect 0x7fcf3c0acfa0 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649286223 0 0) 0x7fcf3c070a20 con 0x7fcf3c072f10 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:38:38.649 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf28004eb0 con 0x7fcf3c072f10 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281741790 0 0) 0x7fcf3c070320 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.659+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c070a20 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 client.?.objecter ms_handle_connect 0x7fcf3c0ad380 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2147717684 0 0) 0x7fcf28004eb0 con 0x7fcf3c072f10 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf3c070320 con 0x7fcf3c072f10 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4099978066 0 0) 0x7fcf3c070a20 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf28004eb0 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcf34006300 con 0x7fcf3c072f10 2024-03-29T10:38:38.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): dump: 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:38.651 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcf2c001a60 con 0x7fcf3c0acfa0 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): dump: 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219182052 0 0) 0x7fcf3c070bb0 con 0x7fcf3c0ad380 2024-03-29T10:38:38.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c070a20 con 0x7fcf3c0ad380 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710040296 0 0) 0x7fcf3c070320 con 0x7fcf3c072f10 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 >> v1:172.21.15.151:6790/0 conn(0x7fcf3c0ad380 legacy=0x7fcf3c071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 >> v1:172.21.15.151:6789/0 conn(0x7fcf3c0acfa0 legacy=0x7fcf3c1db550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 monclient: found mon.b 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c139850 con 0x7fcf3c072f10 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.660+0000 7fcf237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:38.654 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.661+0000 7fcf409a7640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.661+0000 7fcf409a7640 10 monclient: _renew_subs 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.661+0000 7fcf409a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.661+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c139aa0 con 0x7fcf3c072f10 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf409a7640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf409a7640 10 monclient: _renew_subs 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf409a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c1dbe40 con 0x7fcf3c072f10 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf409a7640 1 librados: init done 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf340018c0 con 0x7fcf3c072f10 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fcf34001cd0 con 0x7fcf3c072f10 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:38.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.662+0000 7fcf237fe640 10 monclient: dump: 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:38.656 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.664+0000 7fcf38d99640 4 set_mon_vals no callback set 2024-03-29T10:38:38.657 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.665+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fcf34001eb0 con 0x7fcf3c072f10 2024-03-29T10:38:38.657 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf409a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:38.657 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf409a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:38.657 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf409a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c005080 con 0x7fcf3c072f10 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fcf34084ea0 con 0x7fcf3c072f10 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf237fe640 10 client.4297.objecter ms_dispatch 0x7fcf3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf237fe640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf237fe640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf237fe640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.666+0000 7fcf3a59c640 10 monclient: get_auth_request con 0x7fcf2806dd90 auth_method 0 2024-03-29T10:38:38.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.667+0000 7fcf237fe640 10 client.4297.objecter ms_handle_connect 0x7fcf2806dd90 2024-03-29T10:38:38.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.675+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf34050780 con 0x7fcf3c072f10 2024-03-29T10:38:38.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.675+0000 7fcf237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:38.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:38.675+0000 7fcf237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:38.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.003+0000 7fcf409a7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:38.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.003+0000 7fcf409a7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:38.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.003+0000 7fcf409a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:38:38.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.003+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf1c005330 con 0x7fcf3c072f10 2024-03-29T10:38:38.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.005+0000 7fcf237fe640 1 -- 172.21.15.151:0/1973975447 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fcf340526b0 con 0x7fcf3c072f10 2024-03-29T10:38:38.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.005+0000 7fcf237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:38.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.005+0000 7fcf237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:38.990 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:38.990 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:38.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.009+0000 7fcf409a7640 10 librados: watch_flush enter 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.009+0000 7fcf409a7640 10 librados: watch_flush exit 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.009+0000 7fcf409a7640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.009+0000 7fcf409a7640 10 client.4297.objecter successfully canceled tick 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.010+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 >> v1:172.21.15.151:6816/1549873000 conn(0x7fcf2806dd90 legacy=0x7fcf28070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.010+0000 7fcf409a7640 10 monclient: shutdown 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.010+0000 7fcf409a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:38.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.010+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 >> v1:172.21.15.169:6789/0 conn(0x7fcf3c072f10 legacy=0x7fcf3c139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:38.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.014+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 shutdown_connections 2024-03-29T10:38:38.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.014+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 >> 172.21.15.151:0/1973975447 conn(0x7fcf3c0b3a30 msgr2=0x7fcf3c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:38.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.014+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 shutdown_connections 2024-03-29T10:38:38.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.014+0000 7fcf409a7640 1 -- 172.21.15.151:0/1973975447 wait complete. 2024-03-29T10:38:38.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:39.014+0000 7fcf409a7640 1 librados: shutdown 2024-03-29T10:38:39.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:45.824 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.001+0000 7f01d4a74640 10 monclient: get_monmap_and_config 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.001+0000 7f01d4a74640 10 monclient: build_initial_monmap 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.001+0000 7f01d4a74640 10 monclient: monmap: 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:45.986 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:46.002267+0000 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:46.002267+0000 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:45.987 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:45.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.002+0000 7f01d4a74640 1 Processor -- start 2024-03-29T10:38:45.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.002+0000 7f01d4a74640 1 -- start start 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.002+0000 7f01d4a74640 10 monclient: init 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-b con 0x7f01d00a9930 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-a con 0x7f01d0149dc0 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-c con 0x7f01d014ff40 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00aae60 con 0x7f01d0149dc0 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00b4310 con 0x7f01d014ff40 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00afdd0 con 0x7f01d00a9930 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.003+0000 7f01d4a74640 10 monclient(hunting): authenticate will time out at 1057.173117s 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01ced9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f01d014ff40 0x7f01d0152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:34525/0 (socket says 172.21.15.151:34525) 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01ced9d640 1 -- 172.21.15.151:0/1423378270 learned_addr learned my addr 172.21.15.151:0/1423378270 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032791218 0 0) 0x7f01d00b4310 con 0x7f01d014ff40 2024-03-29T10:38:45.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.007+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b8003520 con 0x7f01d014ff40 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4153067364 0 0) 0x7f01b8003520 con 0x7f01d014ff40 2024-03-29T10:38:45.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 >> v1:172.21.15.151:6790/0 conn(0x7f01d00a9930 legacy=0x7f01d0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:45.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 >> v1:172.21.15.151:6789/0 conn(0x7f01d0149dc0 legacy=0x7f01d014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 monclient: found mon.noname-c 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d014a950 con 0x7f01d014ff40 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: _finish_auth 0 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01d4a74640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01d4a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f01c8001a10 con 0x7f01d014ff40 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:45.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01cd59a640 10 monclient: dump: 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.008+0000 7f01d4a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.009+0000 7f01cd59a640 1 -- 172.21.15.151:0/1423378270 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01c8001ca0 con 0x7f01d014ff40 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.009+0000 7f01cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:45.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.009+0000 7f01d4a74640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.009+0000 7f01d4a74640 4 set_mon_vals no callback set 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.010+0000 7f01d4a74640 10 monclient: shutdown 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.010+0000 7f01d4a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.010+0000 7f01d4a74640 1 -- 172.21.15.151:0/1423378270 >> v1:172.21.15.169:6789/0 conn(0x7f01d014ff40 legacy=0x7f01d0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.011+0000 7f01d4a74640 1 -- 172.21.15.151:0/1423378270 shutdown_connections 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.011+0000 7f01d4a74640 1 -- 172.21.15.151:0/1423378270 >> 172.21.15.151:0/1423378270 conn(0x7f01d00b35e0 msgr2=0x7f01d00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.014+0000 7f01d4a74640 1 -- 172.21.15.151:0/1423378270 shutdown_connections 2024-03-29T10:38:45.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.015+0000 7f01d4a74640 1 -- 172.21.15.151:0/1423378270 wait complete. 2024-03-29T10:38:46.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.016+0000 7f01d4a74640 10 monclient: build_initial_monmap 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.016+0000 7f01d4a74640 10 monclient: monmap: 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.016+0000 7f01d4a74640 1 librados: starting msgr at 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.016+0000 7f01d4a74640 1 librados: starting objecter 2024-03-29T10:38:46.004 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.016+0000 7f01d4a74640 1 Processor -- start 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 1 -- start start 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 1 librados: setting wanted keys 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 1 librados: calling monclient init 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 10 monclient: init 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-b con 0x7f01d00a9930 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.017+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-c con 0x7f01d0149dc0 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 10 monclient(hunting): picked mon.noname-a con 0x7f01d014ff40 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d0149b00 con 0x7f01d014ff40 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00709a0 con 0x7f01d00a9930 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d011cd00 con 0x7f01d0149dc0 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:46.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.018+0000 7f01d4a74640 10 monclient(hunting): authenticate will time out at 1057.188462s 2024-03-29T10:38:46.006 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.020+0000 7f01ced9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f01d014ff40 0x7f01d01e7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:55905/0 (socket says 172.21.15.151:55905) 2024-03-29T10:38:46.006 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.021+0000 7f01ced9d640 1 -- 172.21.15.151:0/1091589784 learned_addr learned my addr 172.21.15.151:0/1091589784 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:46.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 client.?.objecter ms_handle_connect 0x7f01d014ff40 2024-03-29T10:38:46.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:46.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469574753 0 0) 0x7f01d0149b00 con 0x7f01d014ff40 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b0004eb0 con 0x7f01d014ff40 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3074118890 0 0) 0x7f01b0004eb0 con 0x7f01d014ff40 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01d0149b00 con 0x7f01d014ff40 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f01c8004d00 con 0x7f01d014ff40 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): dump: 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:46.009 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.023+0000 7f01beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.024+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 986167945 0 0) 0x7f01d0149b00 con 0x7f01d014ff40 2024-03-29T10:38:46.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.024+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 >> v1:172.21.15.151:6790/0 conn(0x7f01d0149dc0 legacy=0x7f01d01e75a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 >> v1:172.21.15.169:6789/0 conn(0x7f01d00a9930 legacy=0x7f01d01e6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 1 monclient: found mon.a 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d01e5230 con 0x7f01d014ff40 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 monclient: _finish_auth 0 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:46.013 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 10 monclient: _renew_subs 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d01e6f20 con 0x7f01d014ff40 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 10 monclient: _renew_subs 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d0071ee0 con 0x7f01d014ff40 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01d4a74640 1 librados: init done 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 client.4290.objecter ms_handle_connect 0x7f01d00a9930 2024-03-29T10:38:46.014 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.028+0000 7f01beffd640 10 client.4290.objecter resend_mon_ops 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01c8001e90 con 0x7f01d014ff40 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f01c80052f0 con 0x7f01d014ff40 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01beffd640 10 monclient: dump: 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:46.015 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01ccd99640 4 set_mon_vals no callback set 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.029+0000 7f01d4a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.030+0000 7f01d4a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.030+0000 7f01d4a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:46.016 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.030+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b4005080 con 0x7f01d014ff40 2024-03-29T10:38:46.018 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.034+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f01c8005450 con 0x7f01d014ff40 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f01c8085070 con 0x7f01d014ff40 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01beffd640 10 client.4290.objecter ms_dispatch 0x7f01d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01beffd640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01beffd640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01beffd640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T10:38:46.019 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.035+0000 7f01ce59c640 10 monclient: get_auth_request con 0x7f01b006dba0 auth_method 0 2024-03-29T10:38:46.020 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.036+0000 7f01beffd640 10 client.4290.objecter ms_handle_connect 0x7f01b006dba0 2024-03-29T10:38:46.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.039+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01c8050950 con 0x7f01d014ff40 2024-03-29T10:38:46.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.039+0000 7f01beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:46.024 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.040+0000 7f01beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:46.355 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.371+0000 7f01d4a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.371+0000 7f01d4a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.371+0000 7f01d4a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.371+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b4005330 con 0x7f01d014ff40 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.372+0000 7f01beffd640 1 -- 172.21.15.151:0/1091589784 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f01c8052880 con 0x7f01d014ff40 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.372+0000 7f01beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:46.356 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.372+0000 7f01beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:46.357 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:46.357 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:46.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 10 librados: watch_flush enter 2024-03-29T10:38:46.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 10 librados: watch_flush exit 2024-03-29T10:38:46.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T10:38:46.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 10 client.4290.objecter successfully canceled tick 2024-03-29T10:38:46.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 >> v1:172.21.15.151:6816/1549873000 conn(0x7f01b006dba0 legacy=0x7f01b0070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:46.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 10 monclient: shutdown 2024-03-29T10:38:46.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:46.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.377+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 >> v1:172.21.15.151:6789/0 conn(0x7f01d014ff40 legacy=0x7f01d01e7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:46.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.379+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 shutdown_connections 2024-03-29T10:38:46.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.379+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 >> 172.21.15.151:0/1091589784 conn(0x7f01d00b35e0 msgr2=0x7f01d00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:46.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.379+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 shutdown_connections 2024-03-29T10:38:46.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.379+0000 7f01d4a74640 1 -- 172.21.15.151:0/1091589784 wait complete. 2024-03-29T10:38:46.363 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:46.379+0000 7f01d4a74640 1 librados: shutdown 2024-03-29T10:38:46.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:53.191 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.361+0000 7f394d686640 10 monclient: get_monmap_and_config 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.361+0000 7f394d686640 10 monclient: build_initial_monmap 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.361+0000 7f394d686640 10 monclient: monmap: 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:38:53.362800+0000 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:38:53.362800+0000 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:53.348 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:38:53.349 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:38:53.349 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.349 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.365+0000 7f394d686640 1 Processor -- start 2024-03-29T10:38:53.349 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.365+0000 7f394d686640 1 -- start start 2024-03-29T10:38:53.349 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.365+0000 7f394d686640 10 monclient: init 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient(hunting): picked mon.noname-a con 0x7f39480b4100 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient(hunting): picked mon.noname-b con 0x7f394813ee90 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient(hunting): picked mon.noname-c con 0x7f39480a9ac0 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480af6f0 con 0x7f39480b4100 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480aac80 con 0x7f39480a9ac0 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480aaf60 con 0x7f394813ee90 2024-03-29T10:38:53.350 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:53.351 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.366+0000 7f394d686640 10 monclient(hunting): authenticate will time out at 1064.535663s 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.370+0000 7f39467fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f394813ee90 0x7f39480adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60835/0 (socket says 172.21.15.151:60835) 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.370+0000 7f39467fc640 1 -- 172.21.15.151:0/768462121 learned_addr learned my addr 172.21.15.151:0/768462121 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059828928 0 0) 0x7f39480af6f0 con 0x7f39480b4100 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:38:53.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003520 con 0x7f39480b4100 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338662061 0 0) 0x7f39480aaf60 con 0x7f394813ee90 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:38:53.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.373+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39480af6f0 con 0x7f394813ee90 2024-03-29T10:38:53.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339544274 0 0) 0x7f39480aac80 con 0x7f39480a9ac0 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.375+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39480aaf60 con 0x7f39480a9ac0 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83587488 0 0) 0x7f39480aaf60 con 0x7f39480a9ac0 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 >> v1:172.21.15.151:6790/0 conn(0x7f394813ee90 legacy=0x7f39480adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 >> v1:172.21.15.151:6789/0 conn(0x7f39480b4100 legacy=0x7f394813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 monclient: found mon.noname-c 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:38:53.361 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394813e970 con 0x7f39480a9ac0 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f393c001570 con 0x7f39480a9ac0 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.376+0000 7f3945ffb640 10 monclient: dump: 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.362 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:53.363 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:53.363 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:53.363 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.377+0000 7f394d686640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.377+0000 7f394d686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f393c0016d0 con 0x7f39480a9ac0 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f3945ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f394d686640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f394d686640 4 set_mon_vals no callback set 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f3945ffb640 1 -- 172.21.15.151:0/768462121 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f393c001ae0 con 0x7f39480a9ac0 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.378+0000 7f3945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f3945ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f3945ffb640 10 monclient: dump: 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.365 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f394d686640 10 monclient: shutdown 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f394d686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f394d686640 1 -- 172.21.15.151:0/768462121 >> v1:172.21.15.169:6789/0 conn(0x7f39480a9ac0 legacy=0x7f394814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f394d686640 1 -- 172.21.15.151:0/768462121 shutdown_connections 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.379+0000 7f394d686640 1 -- 172.21.15.151:0/768462121 >> 172.21.15.151:0/768462121 conn(0x7f39480b35e0 msgr2=0x7f39480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.380+0000 7f394d686640 1 -- 172.21.15.151:0/768462121 shutdown_connections 2024-03-29T10:38:53.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.380+0000 7f394d686640 1 -- 172.21.15.151:0/768462121 wait complete. 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.381+0000 7f394d686640 10 monclient: build_initial_monmap 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.381+0000 7f394d686640 10 monclient: monmap: 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:53.368 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.381+0000 7f394d686640 1 librados: starting msgr at 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.381+0000 7f394d686640 1 librados: starting objecter 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.381+0000 7f394d686640 1 Processor -- start 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 -- start start 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 librados: setting wanted keys 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 librados: calling monclient init 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient: init 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient(hunting): picked mon.noname-c con 0x7f39480a9ac0 addr v1:172.21.15.151:6790/0 2024-03-29T10:38:53.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient(hunting): picked mon.noname-b con 0x7f39480b4100 addr v1:172.21.15.169:6789/0 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient(hunting): picked mon.noname-a con 0x7f394813ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3948070ce0 con 0x7f394813ee90 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394806f830 con 0x7f39480b4100 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394806f4c0 con 0x7f39480a9ac0 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.382+0000 7f394d686640 10 monclient(hunting): authenticate will time out at 1064.552418s 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.386+0000 7f3946ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f39480a9ac0 0x7f3948131670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60957/0 (socket says 172.21.15.151:60957) 2024-03-29T10:38:53.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.387+0000 7f3946ffd640 1 -- 172.21.15.151:0/3720377360 learned_addr learned my addr 172.21.15.151:0/3720377360 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:38:53.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 client.?.objecter ms_handle_connect 0x7f39480b4100 2024-03-29T10:38:53.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:53.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019170541 0 0) 0x7f394806f830 con 0x7f39480b4100 2024-03-29T10:38:53.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.389+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3934004eb0 con 0x7f39480b4100 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 client.?.objecter ms_handle_connect 0x7f39480a9ac0 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 client.?.objecter ms_handle_connect 0x7f394813ee90 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121912481 0 0) 0x7f394806f4c0 con 0x7f39480a9ac0 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394806f830 con 0x7f39480a9ac0 2024-03-29T10:38:53.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294142619 0 0) 0x7f3934004eb0 con 0x7f39480b4100 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f394806f4c0 con 0x7f39480b4100 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3940001b20 con 0x7f39480b4100 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): dump: 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:53.378 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487366280 0 0) 0x7f394806f830 con 0x7f39480a9ac0 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3934004eb0 con 0x7f39480a9ac0 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3938001950 con 0x7f39480a9ac0 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 10 monclient(hunting): dump: 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.379 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:53.380 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:53.380 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:53.380 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.390+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1662599810 0 0) 0x7f3934004eb0 con 0x7f39480a9ac0 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 >> v1:172.21.15.169:6789/0 conn(0x7f39480b4100 legacy=0x7f3948071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 >> v1:172.21.15.151:6789/0 conn(0x7f394813ee90 legacy=0x7f39481e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 1 monclient: found mon.c 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394806fb10 con 0x7f39480a9ac0 2024-03-29T10:38:53.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f392f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 10 monclient: _renew_subs 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394806fd60 con 0x7f39480a9ac0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 10 monclient: _renew_subs 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3948131f60 con 0x7f39480a9ac0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.391+0000 7f394d686640 1 librados: init done 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3938003610 con 0x7f39480a9ac0 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:53.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f39380039e0 con 0x7f39480a9ac0 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 10 monclient: dump: 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.393+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f3938003c00 con 0x7f39480a9ac0 2024-03-29T10:38:53.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f3944ff9640 4 set_mon_vals no callback set 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f3938084b50 con 0x7f39480a9ac0 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f392f7fe640 10 client.4289.objecter ms_dispatch 0x7f3948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f392f7fe640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f392f7fe640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f392f7fe640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f394d686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.394+0000 7f394d686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.395+0000 7f394d686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.395+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3928005080 con 0x7f39480a9ac0 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.395+0000 7f39467fc640 10 monclient: get_auth_request con 0x7f393406df50 auth_method 0 2024-03-29T10:38:53.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.396+0000 7f392f7fe640 10 client.4289.objecter ms_handle_connect 0x7f393406df50 2024-03-29T10:38:53.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.407+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39380504b0 con 0x7f39480a9ac0 2024-03-29T10:38:53.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.407+0000 7f392f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:53.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.407+0000 7f392f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:53.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.734+0000 7f394d686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:53.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.734+0000 7f394d686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:53.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.734+0000 7f394d686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:38:53.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.734+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3928005330 con 0x7f39480a9ac0 2024-03-29T10:38:53.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.735+0000 7f392f7fe640 1 -- 172.21.15.151:0/3720377360 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f39380523e0 con 0x7f39480a9ac0 2024-03-29T10:38:53.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.735+0000 7f392f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:53.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.735+0000 7f392f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:53.720 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:38:53.720 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:53.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.739+0000 7f394d686640 10 librados: watch_flush enter 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.739+0000 7f394d686640 10 librados: watch_flush exit 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.739+0000 7f394d686640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.739+0000 7f394d686640 10 client.4289.objecter successfully canceled tick 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.740+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 >> v1:172.21.15.151:6816/1549873000 conn(0x7f393406df50 legacy=0x7f3934070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.740+0000 7f394d686640 10 monclient: shutdown 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.740+0000 7f394d686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:53.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.740+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 >> v1:172.21.15.151:6790/0 conn(0x7f39480a9ac0 legacy=0x7f3948131670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.743+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 shutdown_connections 2024-03-29T10:38:53.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.743+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 >> 172.21.15.151:0/3720377360 conn(0x7f39480b35e0 msgr2=0x7f39480b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:53.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.745+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 shutdown_connections 2024-03-29T10:38:53.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.745+0000 7f394d686640 1 -- 172.21.15.151:0/3720377360 wait complete. 2024-03-29T10:38:53.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:38:53.745+0000 7f394d686640 1 librados: shutdown 2024-03-29T10:38:53.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:00.555 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.720+0000 7f600d37b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.720+0000 7f600d37b640 10 monclient: build_initial_monmap 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.720+0000 7f600d37b640 10 monclient: monmap: 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:00.721031+0000 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:00.721031+0000 2024-03-29T10:39:00.704 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 1 Processor -- start 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 1 -- start start 2024-03-29T10:39:00.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient: init 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-c con 0x7f6008071de0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-a con 0x7f60080721c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-b con 0x7f60081425d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600813fb10 con 0x7f60080721c0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6008142c20 con 0x7f6008071de0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60080ada50 con 0x7f60081425d0 2024-03-29T10:39:00.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:00.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.721+0000 7f600d37b640 10 monclient(hunting): authenticate will time out at 1071.891503s 2024-03-29T10:39:00.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.727+0000 7f6006ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f6008071de0 0x7f600813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:34025/0 (socket says 172.21.15.151:34025) 2024-03-29T10:39:00.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.727+0000 7f6006ffd640 1 -- 172.21.15.151:0/1104456345 learned_addr learned my addr 172.21.15.151:0/1104456345 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021665706 0 0) 0x7f6008142c20 con 0x7f6008071de0 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:39:00.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff4003520 con 0x7f6008071de0 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733581977 0 0) 0x7f600813fb10 con 0x7f60080721c0 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:39:00.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.728+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6008142c20 con 0x7f60080721c0 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109572718 0 0) 0x7f6008142c20 con 0x7f60080721c0 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 >> v1:172.21.15.151:6790/0 conn(0x7f60081425d0 legacy=0x7f600814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 >> v1:172.21.15.169:6789/0 conn(0x7f6008071de0 legacy=0x7f600813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 monclient: found mon.noname-a 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60080b0500 con 0x7f60080721c0 2024-03-29T10:39:00.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ff00015a0 con 0x7f60080721c0 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: dump: 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:00.715 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f600d37b640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f600d37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff0001780 con 0x7f60080721c0 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 1 -- 172.21.15.151:0/1104456345 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5ff00015a0 con 0x7f60080721c0 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f6005ffb640 10 monclient: dump: 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:00.716 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f600d37b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:00.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.729+0000 7f600d37b640 4 set_mon_vals no callback set 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.730+0000 7f600d37b640 10 monclient: shutdown 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.730+0000 7f600d37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.730+0000 7f600d37b640 1 -- 172.21.15.151:0/1104456345 >> v1:172.21.15.151:6789/0 conn(0x7f60080721c0 legacy=0x7f6008145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.732+0000 7f600d37b640 1 -- 172.21.15.151:0/1104456345 shutdown_connections 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.732+0000 7f600d37b640 1 -- 172.21.15.151:0/1104456345 >> 172.21.15.151:0/1104456345 conn(0x7f600806b660 msgr2=0x7f600806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.733+0000 7f600d37b640 1 -- 172.21.15.151:0/1104456345 shutdown_connections 2024-03-29T10:39:00.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.734+0000 7f600d37b640 1 -- 172.21.15.151:0/1104456345 wait complete. 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.735+0000 7f600d37b640 10 monclient: build_initial_monmap 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.735+0000 7f600d37b640 10 monclient: monmap: 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:00.719 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.720 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:00.720 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:00.720 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:00.720 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.721 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 librados: starting msgr at 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 librados: starting objecter 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 Processor -- start 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 -- start start 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 librados: setting wanted keys 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 1 librados: calling monclient init 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.736+0000 7f600d37b640 10 monclient: init 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-c con 0x7f6008071de0 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-a con 0x7f60081425d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient(hunting): picked mon.noname-b con 0x7f60081db2d0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60081d75f0 con 0x7f60081425d0 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600813cb30 con 0x7f60081db2d0 2024-03-29T10:39:00.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60081d4fd0 con 0x7f6008071de0 2024-03-29T10:39:00.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:00.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.737+0000 7f600d37b640 10 monclient(hunting): authenticate will time out at 1071.906775s 2024-03-29T10:39:00.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.741+0000 7f60077fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f60081db2d0 0x7f60081e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:34119/0 (socket says 172.21.15.151:34119) 2024-03-29T10:39:00.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.741+0000 7f60077fe640 1 -- 172.21.15.151:0/505140641 learned_addr learned my addr 172.21.15.151:0/505140641 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 client.?.objecter ms_handle_connect 0x7f60081db2d0 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 client.?.objecter ms_handle_connect 0x7f6008071de0 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575184847 0 0) 0x7f600813cb30 con 0x7f60081db2d0 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:00.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.743+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8004eb0 con 0x7f60081db2d0 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391476223 0 0) 0x7f5fe8004eb0 con 0x7f60081db2d0 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600813cb30 con 0x7f60081db2d0 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6000003f60 con 0x7f60081db2d0 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.731 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177669802 0 0) 0x7f60081d4fd0 con 0x7f6008071de0 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8004eb0 con 0x7f6008071de0 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.744+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3825466520 0 0) 0x7f600813cb30 con 0x7f60081db2d0 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 >> v1:172.21.15.151:6790/0 conn(0x7f6008071de0 legacy=0x7f60081d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 >> v1:172.21.15.151:6789/0 conn(0x7f60081425d0 legacy=0x7f60081dabc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:00.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 1 monclient: found mon.b 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60081db780 con 0x7f60081db2d0 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.745+0000 7f5fff7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f5fff7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:00.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f5fff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 10 monclient: _renew_subs 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60081d8190 con 0x7f60081db2d0 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 10 monclient: _renew_subs 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60081e3580 con 0x7f60081db2d0 2024-03-29T10:39:00.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.746+0000 7f600d37b640 1 librados: init done 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.747+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60000040c0 con 0x7f60081db2d0 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.747+0000 7f600d37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.747+0000 7f600d37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.747+0000 7f600d37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.748+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6008075f10 con 0x7f60081db2d0 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.748+0000 7f5fff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.748+0000 7f60057fa640 4 set_mon_vals no callback set 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.749+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6000001070 con 0x7f60081db2d0 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.749+0000 7f5fff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.749+0000 7f5fff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.749+0000 7f5fff7fe640 10 monclient: dump: 2024-03-29T10:39:00.736 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:00.737 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.752+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f60000011d0 con 0x7f60081db2d0 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.752+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f6000084890 con 0x7f60081db2d0 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.752+0000 7f5fff7fe640 10 client.4315.objecter ms_dispatch 0x7f6008000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.753+0000 7f5fff7fe640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.753+0000 7f5fff7fe640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.753+0000 7f5fff7fe640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T10:39:00.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.754+0000 7f6006ffd640 10 monclient: get_auth_request con 0x7f5fe806dcf0 auth_method 0 2024-03-29T10:39:00.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.756+0000 7f5fff7fe640 10 client.4315.objecter ms_handle_connect 0x7f5fe806dcf0 2024-03-29T10:39:00.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.759+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6000050170 con 0x7f60081db2d0 2024-03-29T10:39:00.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.759+0000 7f5fff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:00.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:00.759+0000 7f5fff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:01.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.093+0000 7f600d37b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:01.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.093+0000 7f600d37b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:01.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.093+0000 7f600d37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:01.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.093+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6008145ae0 con 0x7f60081db2d0 2024-03-29T10:39:01.079 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:01.080 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:01.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.094+0000 7f5fff7fe640 1 -- 172.21.15.151:0/505140641 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f60000520a0 con 0x7f60081db2d0 2024-03-29T10:39:01.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.094+0000 7f5fff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:01.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.094+0000 7f5fff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:01.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.098+0000 7f600d37b640 10 librados: watch_flush enter 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 10 librados: watch_flush exit 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 10 client.4315.objecter successfully canceled tick 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 >> v1:172.21.15.151:6816/1549873000 conn(0x7f5fe806dcf0 legacy=0x7f5fe80701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 10 monclient: shutdown 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:01.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.099+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 >> v1:172.21.15.169:6789/0 conn(0x7f60081db2d0 legacy=0x7f60081e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:01.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.100+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 shutdown_connections 2024-03-29T10:39:01.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.100+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 >> 172.21.15.151:0/505140641 conn(0x7f600806b660 msgr2=0x7f60080a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:01.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.100+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 shutdown_connections 2024-03-29T10:39:01.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.102+0000 7f600d37b640 1 -- 172.21.15.151:0/505140641 wait complete. 2024-03-29T10:39:01.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:01.104+0000 7f600d37b640 1 librados: shutdown 2024-03-29T10:39:01.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:07.912 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.087+0000 7f74d738c640 10 monclient: get_monmap_and_config 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.087+0000 7f74d738c640 10 monclient: build_initial_monmap 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.087+0000 7f74d738c640 10 monclient: monmap: 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:08.088068+0000 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:08.088068+0000 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:08.071 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.072 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:08.072 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:08.072 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:08.072 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.088+0000 7f74d738c640 1 Processor -- start 2024-03-29T10:39:08.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.088+0000 7f74d738c640 1 -- start start 2024-03-29T10:39:08.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.088+0000 7f74d738c640 10 monclient: init 2024-03-29T10:39:08.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.089+0000 7f74d738c640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:08.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.089+0000 7f74d738c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:08.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.089+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-b con 0x7f74d0070170 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:08.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.089+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-a con 0x7f74d00713f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d5128640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f74d0070170 0x7f74d013d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:51150/0 (socket says 172.21.15.151:51150) 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-c con 0x7f74d0070550 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d5128640 1 -- 172.21.15.151:0/1359856606 learned_addr learned my addr 172.21.15.151:0/1359856606 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d013dbd0 con 0x7f74d00713f0 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d013ff70 con 0x7f74d0070550 2024-03-29T10:39:08.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0140250 con 0x7f74d0070170 2024-03-29T10:39:08.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:08.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.093+0000 7f74d738c640 10 monclient(hunting): authenticate will time out at 1079.263196s 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502600684 0 0) 0x7f74d013ff70 con 0x7f74d0070550 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:39:08.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.095+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc003520 con 0x7f74d0070550 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341538021 0 0) 0x7f74c0001110 con 0x7f74d00713f0 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d013ff70 con 0x7f74d00713f0 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2829619504 0 0) 0x7f74bc003520 con 0x7f74d0070550 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 >> v1:172.21.15.151:6790/0 conn(0x7f74d0070170 legacy=0x7f74d013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 >> v1:172.21.15.151:6789/0 conn(0x7f74d00713f0 legacy=0x7f74d01406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.096+0000 7f74c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0140df0 con 0x7f74d0070550 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f74c8001990 con 0x7f74d0070550 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:08.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: dump: 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74d738c640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:39:08.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74d738c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74c8001af0 con 0x7f74d0070550 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 1 -- 172.21.15.151:0/1359856606 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f74c8001f40 con 0x7f74d0070550 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74c7fff640 10 monclient: dump: 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.091 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74d738c640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.097+0000 7f74d738c640 4 set_mon_vals no callback set 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.098+0000 7f74d738c640 10 monclient: shutdown 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.098+0000 7f74d738c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.098+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 >> v1:172.21.15.169:6789/0 conn(0x7f74d0070550 legacy=0x7f74d014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.099+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 shutdown_connections 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.099+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 >> 172.21.15.151:0/1359856606 conn(0x7f74d0067610 msgr2=0x7f74d00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.100+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 shutdown_connections 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.100+0000 7f74d738c640 1 -- 172.21.15.151:0/1359856606 wait complete. 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.101+0000 7f74d738c640 10 monclient: build_initial_monmap 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.101+0000 7f74d738c640 10 monclient: monmap: 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:08.093 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 librados: starting msgr at 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 librados: starting objecter 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 Processor -- start 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 -- start start 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 librados: setting wanted keys 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 1 librados: calling monclient init 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.102+0000 7f74d738c640 10 monclient: init 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-b con 0x7f74d0070170 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-a con 0x7f74d0070550 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient(hunting): picked mon.noname-c con 0x7f74d00713f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d014b9e0 con 0x7f74d0070550 2024-03-29T10:39:08.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0131550 con 0x7f74d0070170 2024-03-29T10:39:08.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d014bc50 con 0x7f74d00713f0 2024-03-29T10:39:08.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:08.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d738c640 10 monclient(hunting): authenticate will time out at 1079.273430s 2024-03-29T10:39:08.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.103+0000 7f74d5929640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f74d00713f0 0x7f74d0130e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:51712/0 (socket says 172.21.15.151:51712) 2024-03-29T10:39:08.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.104+0000 7f74d5929640 1 -- 172.21.15.151:0/3540896431 learned_addr learned my addr 172.21.15.151:0/3540896431 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 client.?.objecter ms_handle_connect 0x7f74d00713f0 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 client.?.objecter ms_handle_connect 0x7f74d0070170 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75597219 0 0) 0x7f74d014bc50 con 0x7f74d00713f0 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:08.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b4004eb0 con 0x7f74d00713f0 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924112576 0 0) 0x7f74d0131550 con 0x7f74d0070170 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.107+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d014bc50 con 0x7f74d0070170 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2859582759 0 0) 0x7f74b4004eb0 con 0x7f74d00713f0 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74d0131550 con 0x7f74d00713f0 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f74c8001ec0 con 0x7f74d00713f0 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:08.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:08.104 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1543206586 0 0) 0x7f74d014bc50 con 0x7f74d0070170 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b4004eb0 con 0x7f74d0070170 2024-03-29T10:39:08.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f74cc001740 con 0x7f74d0070170 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:08.106 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2842994759 0 0) 0x7f74d0131550 con 0x7f74d00713f0 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 >> v1:172.21.15.169:6789/0 conn(0x7f74d0070170 legacy=0x7f74d01dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 >> v1:172.21.15.151:6789/0 conn(0x7f74d0070550 legacy=0x7f74d0130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 monclient: found mon.c 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.108+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d01df150 con 0x7f74d00713f0 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:08.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74d738c640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74d738c640 10 monclient: _renew_subs 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74d738c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.109+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d014a390 con 0x7f74d00713f0 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74d738c640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74d738c640 10 monclient: _renew_subs 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74d738c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d014a040 con 0x7f74d00713f0 2024-03-29T10:39:08.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74c8003c20 con 0x7f74d00713f0 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74d738c640 1 librados: init done 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.110+0000 7f74c77fe640 4 set_mon_vals no callback set 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.111+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f74c8004030 con 0x7f74d00713f0 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.111+0000 7f74c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.111+0000 7f74c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.111+0000 7f74c5ffb640 10 monclient: dump: 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:08.109 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.113+0000 7f74d738c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.113+0000 7f74d738c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.113+0000 7f74d738c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.113+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b0005080 con 0x7f74d00713f0 2024-03-29T10:39:08.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f74c8004190 con 0x7f74d00713f0 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f74c8084da0 con 0x7f74d00713f0 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 10 client.4298.objecter ms_dispatch 0x7f74d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.115+0000 7f74c5ffb640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.116+0000 7f74d5128640 10 monclient: get_auth_request con 0x7f74b406de10 auth_method 0 2024-03-29T10:39:08.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.116+0000 7f74c5ffb640 10 client.4298.objecter ms_handle_connect 0x7f74b406de10 2024-03-29T10:39:08.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.125+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74c8050680 con 0x7f74d00713f0 2024-03-29T10:39:08.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.125+0000 7f74c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:08.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.125+0000 7f74c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:08.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.486+0000 7f74d738c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:08.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.486+0000 7f74d738c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:08.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.486+0000 7f74d738c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:08.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.486+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b0005330 con 0x7f74d00713f0 2024-03-29T10:39:08.471 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:08.472 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:08.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.487+0000 7f74c5ffb640 1 -- 172.21.15.151:0/3540896431 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f74c80525b0 con 0x7f74d00713f0 2024-03-29T10:39:08.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.487+0000 7f74c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:08.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.487+0000 7f74c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:08.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 10 librados: watch_flush enter 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 10 librados: watch_flush exit 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 >> v1:172.21.15.151:6816/1549873000 conn(0x7f74b406de10 legacy=0x7f74b40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 10 monclient: shutdown 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.491+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 >> v1:172.21.15.151:6790/0 conn(0x7f74d00713f0 legacy=0x7f74d0130e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.492+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 shutdown_connections 2024-03-29T10:39:08.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.492+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 >> 172.21.15.151:0/3540896431 conn(0x7f74d0067610 msgr2=0x7f74d00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:08.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.492+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 shutdown_connections 2024-03-29T10:39:08.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.492+0000 7f74d738c640 1 -- 172.21.15.151:0/3540896431 wait complete. 2024-03-29T10:39:08.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:08.492+0000 7f74d738c640 1 librados: shutdown 2024-03-29T10:39:08.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:15.300 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.468+0000 7f9671012640 10 monclient: get_monmap_and_config 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.468+0000 7f9671012640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.468+0000 7f9671012640 10 monclient: monmap: 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.452 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:15.469222+0000 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:15.469222+0000 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.453 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.469+0000 7f9671012640 1 Processor -- start 2024-03-29T10:39:15.454 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.469+0000 7f9671012640 1 -- start start 2024-03-29T10:39:15.454 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.469+0000 7f9671012640 10 monclient: init 2024-03-29T10:39:15.454 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.454 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:15.454 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient(hunting): picked mon.noname-c con 0x7f966c0b2800 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient(hunting): picked mon.noname-b con 0x7f966c0a9c00 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient(hunting): picked mon.noname-a con 0x7f966c0b38c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c141eb0 con 0x7f966c0b38c0 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c0b22d0 con 0x7f966c0b2800 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f966bfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f966c0b2800 0x7f966c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:38480/0 (socket says 172.21.15.151:38480) 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c142d30 con 0x7f966c0a9c00 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f966bfff640 1 -- 172.21.15.151:0/3448448467 learned_addr learned my addr 172.21.15.151:0/3448448467 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.470+0000 7f9671012640 10 monclient(hunting): authenticate will time out at 1086.640464s 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142158757 0 0) 0x7f966c142d30 con 0x7f966c0a9c00 2024-03-29T10:39:15.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9654003520 con 0x7f966c0a9c00 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798256468 0 0) 0x7f966c0b22d0 con 0x7f966c0b2800 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c142d30 con 0x7f966c0b2800 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935984698 0 0) 0x7f966c141eb0 con 0x7f966c0b38c0 2024-03-29T10:39:15.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c0b22d0 con 0x7f966c0b38c0 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.471+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434547611 0 0) 0x7f966c142d30 con 0x7f966c0b2800 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 >> v1:172.21.15.151:6790/0 conn(0x7f966c0a9c00 legacy=0x7f966c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 >> v1:172.21.15.151:6789/0 conn(0x7f966c0b38c0 legacy=0x7f966c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 monclient: found mon.noname-c 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0aaf90 con 0x7f966c0b2800 2024-03-29T10:39:15.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f9671012640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f9671012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f965c001670 con 0x7f966c0b2800 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: dump: 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.459 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f9671012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f965c001850 con 0x7f966c0b2800 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f9671012640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f9671012640 4 set_mon_vals no callback set 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 1 -- 172.21.15.151:0/3448448467 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f965c001670 con 0x7f966c0b2800 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.472+0000 7f966affd640 10 monclient: dump: 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:15.461 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 10 monclient: shutdown 2024-03-29T10:39:15.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 1 -- 172.21.15.151:0/3448448467 >> v1:172.21.15.169:6789/0 conn(0x7f966c0b2800 legacy=0x7f966c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 1 -- 172.21.15.151:0/3448448467 shutdown_connections 2024-03-29T10:39:15.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 1 -- 172.21.15.151:0/3448448467 >> 172.21.15.151:0/3448448467 conn(0x7f966c066a00 msgr2=0x7f966c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 1 -- 172.21.15.151:0/3448448467 shutdown_connections 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.473+0000 7f9671012640 1 -- 172.21.15.151:0/3448448467 wait complete. 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.474+0000 7f9671012640 10 monclient: build_initial_monmap 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.474+0000 7f9671012640 10 monclient: monmap: 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.466 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.474+0000 7f9671012640 1 librados: starting msgr at 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.474+0000 7f9671012640 1 librados: starting objecter 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.474+0000 7f9671012640 1 Processor -- start 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.475+0000 7f9671012640 1 -- start start 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.475+0000 7f9671012640 1 librados: setting wanted keys 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.475+0000 7f9671012640 1 librados: calling monclient init 2024-03-29T10:39:15.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.475+0000 7f9671012640 10 monclient: init 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient(hunting): picked mon.noname-a con 0x7f966c0a9c00 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient(hunting): picked mon.noname-c con 0x7f966c0b2800 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient(hunting): picked mon.noname-b con 0x7f966c0b38c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f966b7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f966c0b2800 0x7f966c143960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:54592/0 (socket says 172.21.15.151:54592) 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f966b7fe640 1 -- 172.21.15.151:0/4060501280 learned_addr learned my addr 172.21.15.151:0/4060501280 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c1431f0 con 0x7f966c0a9c00 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c06f720 con 0x7f966c0b38c0 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c06fa00 con 0x7f966c0b2800 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9671012640 10 monclient(hunting): authenticate will time out at 1086.646051s 2024-03-29T10:39:15.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter ms_handle_connect 0x7f966c0b2800 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter ms_handle_connect 0x7f966c0a9c00 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter ms_handle_connect 0x7f966c0b38c0 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973071465 0 0) 0x7f966c06fa00 con 0x7f966c0b2800 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:39:15.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.476+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9658004eb0 con 0x7f966c0b2800 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776246099 0 0) 0x7f966c06f720 con 0x7f966c0b38c0 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c06fa00 con 0x7f966c0b38c0 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163661347 0 0) 0x7f966c06fa00 con 0x7f966c0b38c0 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f966c06f720 con 0x7f966c0b38c0 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9664001b50 con 0x7f966c0b38c0 2024-03-29T10:39:15.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:15.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913584292 0 0) 0x7f966c1431f0 con 0x7f966c0a9c00 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c06fa00 con 0x7f966c0a9c00 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760427009 0 0) 0x7f9658004eb0 con 0x7f966c0b2800 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f966c1431f0 con 0x7f966c0b2800 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9660001950 con 0x7f966c0b2800 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.477+0000 7f9668ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901529888 0 0) 0x7f966c06f720 con 0x7f966c0b38c0 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 >> v1:172.21.15.151:6790/0 conn(0x7f966c0b2800 legacy=0x7f966c143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 >> v1:172.21.15.151:6789/0 conn(0x7f966c0a9c00 legacy=0x7f966c1db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 monclient: found mon.b 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c06de90 con 0x7f966c0b38c0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 10 monclient: _renew_subs 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c06e140 con 0x7f966c0b38c0 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:15.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 10 monclient: _renew_subs 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c146060 con 0x7f966c0b38c0 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9671012640 1 librados: init done 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9664001fe0 con 0x7f966c0b38c0 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9664003b30 con 0x7f966c0b38c0 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f9668ff9640 10 monclient: dump: 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:15.475 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.478+0000 7f966a7fc640 4 set_mon_vals no callback set 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.480+0000 7f9671012640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.480+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f9664003c90 con 0x7f966c0b38c0 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.480+0000 7f9671012640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.480+0000 7f9671012640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.480+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c005080 con 0x7f966c0b38c0 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f9664083b40 con 0x7f966c0b38c0 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 10 client.4327.objecter ms_dispatch 0x7f966c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:15.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T10:39:15.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f966bfff640 10 monclient: get_auth_request con 0x7f965806dfd0 auth_method 0 2024-03-29T10:39:15.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.481+0000 7f9668ff9640 10 client.4327.objecter ms_handle_connect 0x7f965806dfd0 2024-03-29T10:39:15.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.487+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f966404f420 con 0x7f966c0b38c0 2024-03-29T10:39:15.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.487+0000 7f9668ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:15.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.487+0000 7f9668ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:15.830 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.846+0000 7f9671012640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.846+0000 7f9671012640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.846+0000 7f9671012640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.846+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f964c005330 con 0x7f966c0b38c0 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.847+0000 7f9668ff9640 1 -- 172.21.15.151:0/4060501280 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f9664051350 con 0x7f966c0b38c0 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.847+0000 7f9668ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.847+0000 7f9668ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:15.831 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:15.832 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:15.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 10 librados: watch_flush enter 2024-03-29T10:39:15.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 10 librados: watch_flush exit 2024-03-29T10:39:15.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T10:39:15.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 10 client.4327.objecter successfully canceled tick 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 >> v1:172.21.15.151:6816/1549873000 conn(0x7f965806dfd0 legacy=0x7f9658070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 10 monclient: shutdown 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 >> v1:172.21.15.169:6789/0 conn(0x7f966c0b38c0 legacy=0x7f966c144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 shutdown_connections 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.851+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 >> 172.21.15.151:0/4060501280 conn(0x7f966c066a00 msgr2=0x7f966c140590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.852+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 shutdown_connections 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.852+0000 7f9671012640 1 -- 172.21.15.151:0/4060501280 wait complete. 2024-03-29T10:39:15.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:15.852+0000 7f9671012640 1 librados: shutdown 2024-03-29T10:39:15.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:22.658 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.805+0000 7fbf30682640 10 monclient: get_monmap_and_config 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.805+0000 7fbf30682640 10 monclient: build_initial_monmap 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.805+0000 7fbf30682640 10 monclient: monmap: 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:22.807887+0000 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:22.807887+0000 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:22.791 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.807+0000 7fbf30682640 1 Processor -- start 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.807+0000 7fbf30682640 1 -- start start 2024-03-29T10:39:22.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.807+0000 7fbf30682640 10 monclient: init 2024-03-29T10:39:22.793 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-a con 0x7fbf281457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-c con 0x7fbf2813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-b con 0x7fbf280a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf280aaef0 con 0x7fbf281457f0 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28145bd0 con 0x7fbf2813f650 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28145eb0 con 0x7fbf280a9d70 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf30682640 10 monclient(hunting): authenticate will time out at 1093.979076s 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf2ec1f640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fbf280a9d70 0x7fbf2814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:43612/0 (socket says 172.21.15.151:43612) 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.808+0000 7fbf2ec1f640 1 -- 172.21.15.151:0/2085366598 learned_addr learned my addr 172.21.15.151:0/2085366598 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:22.794 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842174817 0 0) 0x7fbf28145eb0 con 0x7fbf280a9d70 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 10 monclient(hunting): _negotiate 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf14003520 con 0x7fbf280a9d70 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262135611 0 0) 0x7fbf14003520 con 0x7fbf280a9d70 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 >> v1:172.21.15.169:6789/0 conn(0x7fbf2813f650 legacy=0x7fbf280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:22.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.809+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 >> v1:172.21.15.151:6789/0 conn(0x7fbf281457f0 legacy=0x7fbf2813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:22.796 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: _finish_hunting 0 2024-03-29T10:39:22.796 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 1 monclient: found mon.noname-b 2024-03-29T10:39:22.796 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:22.796 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:39:22.796 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28146340 con 0x7fbf280a9d70 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: _finish_auth 0 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: _check_auth_tickets 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbf20001a10 con 0x7fbf280a9d70 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: dump: 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:22.797 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf30682640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf30682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf20001c70 con 0x7fbf280a9d70 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 1 -- 172.21.15.151:0/2085366598 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbf200020c0 con 0x7fbf280a9d70 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf2d41c640 10 monclient: dump: 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:22.798 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf30682640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:22.799 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.811+0000 7fbf30682640 4 set_mon_vals no callback set 2024-03-29T10:39:22.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.812+0000 7fbf30682640 10 monclient: shutdown 2024-03-29T10:39:22.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.812+0000 7fbf30682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:22.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.812+0000 7fbf30682640 1 -- 172.21.15.151:0/2085366598 >> v1:172.21.15.151:6790/0 conn(0x7fbf280a9d70 legacy=0x7fbf2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:22.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.813+0000 7fbf30682640 1 -- 172.21.15.151:0/2085366598 shutdown_connections 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.813+0000 7fbf30682640 1 -- 172.21.15.151:0/2085366598 >> 172.21.15.151:0/2085366598 conn(0x7fbf28066a00 msgr2=0x7fbf28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.813+0000 7fbf30682640 1 -- 172.21.15.151:0/2085366598 shutdown_connections 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.813+0000 7fbf30682640 1 -- 172.21.15.151:0/2085366598 wait complete. 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 10 monclient: build_initial_monmap 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 10 monclient: monmap: 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:22.801 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 librados: starting msgr at 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 librados: starting objecter 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 Processor -- start 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 -- start start 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 librados: setting wanted keys 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 1 librados: calling monclient init 2024-03-29T10:39:22.802 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.815+0000 7fbf30682640 10 monclient: init 2024-03-29T10:39:22.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:22.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:22.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-c con 0x7fbf280a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:22.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf2e41e640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fbf280a9d70 0x7fbf28136bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45148/0 (socket says 172.21.15.151:45148) 2024-03-29T10:39:22.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf2e41e640 1 -- 172.21.15.151:0/3528140408 learned_addr learned my addr 172.21.15.151:0/3528140408 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient(hunting): picked mon.noname-a con 0x7fbf281457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28132d00 con 0x7fbf281457f0 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 client.?.objecter ms_handle_connect 0x7fbf280a9d70 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28137300 con 0x7fbf2813f650 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28135230 con 0x7fbf280a9d70 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf30682640 10 monclient(hunting): authenticate will time out at 1093.987355s 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123128330 0 0) 0x7fbf28135230 con 0x7fbf280a9d70 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:22.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.816+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c004eb0 con 0x7fbf280a9d70 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 client.?.objecter ms_handle_connect 0x7fbf2813f650 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962440434 0 0) 0x7fbf0c004eb0 con 0x7fbf280a9d70 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf28135230 con 0x7fbf280a9d70 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbf240017d0 con 0x7fbf280a9d70 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 monclient(hunting): dump: 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.807 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.817+0000 7fbf1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.818+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 478986814 0 0) 0x7fbf28135230 con 0x7fbf280a9d70 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.818+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 >> v1:172.21.15.169:6789/0 conn(0x7fbf2813f650 legacy=0x7fbf28134b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.818+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 >> v1:172.21.15.151:6789/0 conn(0x7fbf281457f0 legacy=0x7fbf281e2000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.819+0000 7fbf1effd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.819+0000 7fbf1effd640 1 monclient: found mon.c 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf281353c0 con 0x7fbf280a9d70 2024-03-29T10:39:22.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: _finish_auth 0 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 10 monclient: _renew_subs 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf28133e80 con 0x7fbf280a9d70 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 10 monclient: _renew_subs 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf28133400 con 0x7fbf280a9d70 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf30682640 1 librados: init done 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf24001d60 con 0x7fbf280a9d70 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:22.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbf24004a00 con 0x7fbf280a9d70 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf1effd640 10 monclient: dump: 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:22.810 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.820+0000 7fbf2cc1b640 4 set_mon_vals no callback set 2024-03-29T10:39:22.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.822+0000 7fbf30682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:22.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.822+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fbf24004b60 con 0x7fbf280a9d70 2024-03-29T10:39:22.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.822+0000 7fbf30682640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:22.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.822+0000 7fbf30682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:22.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.822+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf10005080 con 0x7fbf280a9d70 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fbf24084970 con 0x7fbf280a9d70 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 10 client.4310.objecter ms_dispatch 0x7fbf28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf2dc1d640 10 monclient: get_auth_request con 0x7fbf0c06db60 auth_method 0 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:39:22.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.823+0000 7fbf1effd640 10 client.4310.objecter ms_handle_connect 0x7fbf0c06db60 2024-03-29T10:39:22.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.830+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf240502b0 con 0x7fbf280a9d70 2024-03-29T10:39:22.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.830+0000 7fbf1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:22.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:22.830+0000 7fbf1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.150+0000 7fbf30682640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.150+0000 7fbf30682640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.151+0000 7fbf30682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.151+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf10005330 con 0x7fbf280a9d70 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.152+0000 7fbf1effd640 1 -- 172.21.15.151:0/3528140408 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fbf240521e0 con 0x7fbf280a9d70 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.152+0000 7fbf1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:23.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.152+0000 7fbf1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:23.137 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:23.137 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:23.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.155+0000 7fbf30682640 10 librados: watch_flush enter 2024-03-29T10:39:23.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 10 librados: watch_flush exit 2024-03-29T10:39:23.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:39:23.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:39:23.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 >> v1:172.21.15.151:6816/1549873000 conn(0x7fbf0c06db60 legacy=0x7fbf0c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:23.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 10 monclient: shutdown 2024-03-29T10:39:23.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:23.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 >> v1:172.21.15.151:6790/0 conn(0x7fbf280a9d70 legacy=0x7fbf28136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:23.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 shutdown_connections 2024-03-29T10:39:23.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 >> 172.21.15.151:0/3528140408 conn(0x7fbf28066a00 msgr2=0x7fbf280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:23.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.156+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 shutdown_connections 2024-03-29T10:39:23.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.157+0000 7fbf30682640 1 -- 172.21.15.151:0/3528140408 wait complete. 2024-03-29T10:39:23.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:23.157+0000 7fbf30682640 1 librados: shutdown 2024-03-29T10:39:23.158 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:29.966 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:30.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.130+0000 7f4b35993640 10 monclient: get_monmap_and_config 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.130+0000 7f4b35993640 10 monclient: build_initial_monmap 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.130+0000 7f4b35993640 10 monclient: monmap: 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:30.132000+0000 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:30.132000+0000 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:30.116 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.130+0000 7f4b35993640 1 Processor -- start 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 1 -- start start 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 10 monclient: init 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-b con 0x7f4b300a9ac0 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:30.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.131+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3013ee90 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:30.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-a con 0x7f4b300af900 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:30.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300afd10 con 0x7f4b300af900 2024-03-29T10:39:30.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300af6f0 con 0x7f4b3013ee90 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b300aac80 con 0x7f4b300a9ac0 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.133+0000 7f4b35993640 10 monclient(hunting): authenticate will time out at 1101.304457s 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.134+0000 7f4b2effd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f4b300a9ac0 0x7f4b300adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56975/0 (socket says 172.21.15.151:56975) 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.134+0000 7f4b2effd640 1 -- 172.21.15.151:0/3597156747 learned_addr learned my addr 172.21.15.151:0/3597156747 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:30.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619945342 0 0) 0x7f4b300aac80 con 0x7f4b300a9ac0 2024-03-29T10:39:30.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:30.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:30.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:30.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:39:30.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.135+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18003520 con 0x7f4b300a9ac0 2024-03-29T10:39:30.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.136+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562085153 0 0) 0x7f4b18003520 con 0x7f4b300a9ac0 2024-03-29T10:39:30.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.136+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 >> v1:172.21.15.169:6789/0 conn(0x7f4b3013ee90 legacy=0x7f4b3013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.136+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 >> v1:172.21.15.151:6789/0 conn(0x7f4b300af900 legacy=0x7f4b30149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 1 monclient: found mon.noname-b 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300aaf60 con 0x7f4b300a9ac0 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:30.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4b20001690 con 0x7f4b300a9ac0 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b2dffb640 10 monclient: dump: 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:30.127 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b35993640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.137+0000 7f4b35993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b20001860 con 0x7f4b300a9ac0 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 1 -- 172.21.15.151:0/3597156747 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4b20001cb0 con 0x7f4b300a9ac0 2024-03-29T10:39:30.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b2dffb640 10 monclient: dump: 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:30.129 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b35993640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.138+0000 7f4b35993640 4 set_mon_vals no callback set 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 10 monclient: shutdown 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 1 -- 172.21.15.151:0/3597156747 >> v1:172.21.15.151:6790/0 conn(0x7f4b300a9ac0 legacy=0x7f4b300adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 1 -- 172.21.15.151:0/3597156747 shutdown_connections 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 1 -- 172.21.15.151:0/3597156747 >> 172.21.15.151:0/3597156747 conn(0x7f4b300b35e0 msgr2=0x7f4b300b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:30.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 1 -- 172.21.15.151:0/3597156747 shutdown_connections 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.139+0000 7f4b35993640 1 -- 172.21.15.151:0/3597156747 wait complete. 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.140+0000 7f4b35993640 10 monclient: build_initial_monmap 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.140+0000 7f4b35993640 10 monclient: monmap: 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:30.131 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.140+0000 7f4b35993640 1 librados: starting msgr at 2024-03-29T10:39:30.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.140+0000 7f4b35993640 1 librados: starting objecter 2024-03-29T10:39:30.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.141+0000 7f4b35993640 1 Processor -- start 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.141+0000 7f4b35993640 1 -- start start 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.141+0000 7f4b35993640 1 librados: setting wanted keys 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.141+0000 7f4b35993640 1 librados: calling monclient init 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.141+0000 7f4b35993640 10 monclient: init 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.142+0000 7f4b35993640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.142+0000 7f4b35993640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.142+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-b con 0x7f4b300a9ac0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.142+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-a con 0x7f4b300af900 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3013ee90 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b30142660 con 0x7f4b300af900 2024-03-29T10:39:30.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b30142310 con 0x7f4b300a9ac0 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b30139230 con 0x7f4b3013ee90 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.144+0000 7f4b35993640 10 monclient(hunting): authenticate will time out at 1101.315239s 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.145+0000 7f4b2effd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4b300a9ac0 0x7f4b30141590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:42268/0 (socket says 172.21.15.151:42268) 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.145+0000 7f4b2effd640 1 -- 172.21.15.151:0/626554079 learned_addr learned my addr 172.21.15.151:0/626554079 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 client.?.objecter ms_handle_connect 0x7f4b300a9ac0 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 client.?.objecter ms_handle_connect 0x7f4b3013ee90 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305192833 0 0) 0x7f4b30139230 con 0x7f4b3013ee90 2024-03-29T10:39:30.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.146+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1c004eb0 con 0x7f4b3013ee90 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.148+0000 7f4b17fff640 10 client.?.objecter ms_handle_connect 0x7f4b300af900 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.148+0000 7f4b17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.148+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549451534 0 0) 0x7f4b1c004eb0 con 0x7f4b3013ee90 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b30139230 con 0x7f4b3013ee90 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627627953 0 0) 0x7f4b30142310 con 0x7f4b300a9ac0 2024-03-29T10:39:30.137 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1c004eb0 con 0x7f4b300a9ac0 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467136832 0 0) 0x7f4b30142660 con 0x7f4b300af900 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b30142310 con 0x7f4b300af900 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4b28001890 con 0x7f4b3013ee90 2024-03-29T10:39:30.138 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): dump: 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:30.139 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:30.140 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4026409147 0 0) 0x7f4b30139230 con 0x7f4b3013ee90 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 >> v1:172.21.15.169:6789/0 conn(0x7f4b300a9ac0 legacy=0x7f4b30141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 >> v1:172.21.15.151:6789/0 conn(0x7f4b300af900 legacy=0x7f4b30138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.149+0000 7f4b17fff640 1 monclient: found mon.c 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.143 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b30140cd0 con 0x7f4b3013ee90 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b28001de0 con 0x7f4b3013ee90 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.150+0000 7f4b2d7fa640 4 set_mon_vals no callback set 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.151+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4b28003c10 con 0x7f4b3013ee90 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b17fff640 10 monclient: dump: 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:30.144 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 10 monclient: _renew_subs 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3013d850 con 0x7f4b3013ee90 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 10 monclient: _renew_subs 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b301db940 con 0x7f4b3013ee90 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.152+0000 7f4b35993640 1 librados: init done 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.154+0000 7f4b35993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.154+0000 7f4b35993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.154+0000 7f4b35993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.154+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3006f1b0 con 0x7f4b3013ee90 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.155+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4b28019be0 con 0x7f4b3013ee90 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.156+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4b28083b10 con 0x7f4b3013ee90 2024-03-29T10:39:30.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.156+0000 7f4b17fff640 10 client.4316.objecter ms_dispatch 0x7f4b30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:30.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.156+0000 7f4b17fff640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:30.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.156+0000 7f4b17fff640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:30.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.156+0000 7f4b17fff640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T10:39:30.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.157+0000 7f4b2effd640 10 monclient: get_auth_request con 0x7f4b1c06dd10 auth_method 0 2024-03-29T10:39:30.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.158+0000 7f4b17fff640 10 client.4316.objecter ms_handle_connect 0x7f4b1c06dd10 2024-03-29T10:39:30.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.163+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b3006f1b0 con 0x7f4b3013ee90 2024-03-29T10:39:30.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.163+0000 7f4b17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:30.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.163+0000 7f4b17fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:30.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.496+0000 7f4b35993640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:30.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.496+0000 7f4b35993640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:30.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.496+0000 7f4b35993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:30.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.496+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b3006f1b0 con 0x7f4b3013ee90 2024-03-29T10:39:30.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.497+0000 7f4b17fff640 1 -- 172.21.15.151:0/626554079 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4b3006f1b0 con 0x7f4b3013ee90 2024-03-29T10:39:30.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.497+0000 7f4b17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:30.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.497+0000 7f4b17fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:30.482 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:30.482 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:30.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 10 librados: watch_flush enter 2024-03-29T10:39:30.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 10 librados: watch_flush exit 2024-03-29T10:39:30.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 10 client.4316.objecter successfully canceled tick 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4b1c06dd10 legacy=0x7f4b1c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 10 monclient: shutdown 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.500+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 >> v1:172.21.15.151:6790/0 conn(0x7f4b3013ee90 legacy=0x7f4b301db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.501+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 shutdown_connections 2024-03-29T10:39:30.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.501+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 >> 172.21.15.151:0/626554079 conn(0x7f4b300b35e0 msgr2=0x7f4b3013b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:30.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.501+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 shutdown_connections 2024-03-29T10:39:30.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.501+0000 7f4b35993640 1 -- 172.21.15.151:0/626554079 wait complete. 2024-03-29T10:39:30.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:30.501+0000 7f4b35993640 1 librados: shutdown 2024-03-29T10:39:30.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:37.309 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:37.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.474+0000 7fc98c932640 10 monclient: get_monmap_and_config 2024-03-29T10:39:37.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.474+0000 7fc98c932640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.474+0000 7fc98c932640 10 monclient: monmap: 2024-03-29T10:39:37.458 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:37.475317+0000 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:37.475317+0000 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:37.459 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:37.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.478+0000 7fc98c932640 1 Processor -- start 2024-03-29T10:39:37.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.478+0000 7fc98c932640 1 -- start start 2024-03-29T10:39:37.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.478+0000 7fc98c932640 10 monclient: init 2024-03-29T10:39:37.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:37.464 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-b con 0x7fc988149b50 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-a con 0x7fc9880b2620 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-c con 0x7fc98814fc70 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.479+0000 7fc98c932640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98806eee0 con 0x7fc9880b2620 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.480+0000 7fc98c932640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98806f1c0 con 0x7fc98814fc70 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.480+0000 7fc98c932640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc988071d80 con 0x7fc988149b50 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.480+0000 7fc98c932640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.480+0000 7fc98c932640 10 monclient(hunting): authenticate will time out at 1108.649541s 2024-03-29T10:39:37.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.482+0000 7fc985d9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc9880b2620 0x7fc9880b2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59151/0 (socket says 172.21.15.151:59151) 2024-03-29T10:39:37.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.482+0000 7fc985d9b640 1 -- 172.21.15.151:0/402308306 learned_addr learned my addr 172.21.15.151:0/402308306 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205274411 0 0) 0x7fc98806eee0 con 0x7fc9880b2620 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:39:37.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.483+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc970003520 con 0x7fc9880b2620 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640403604 0 0) 0x7fc988071d80 con 0x7fc988149b50 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98806eee0 con 0x7fc988149b50 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032219299 0 0) 0x7fc98806f1c0 con 0x7fc98814fc70 2024-03-29T10:39:37.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:39:37.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.484+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc988071d80 con 0x7fc98814fc70 2024-03-29T10:39:37.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.486+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3134670522 0 0) 0x7fc98806eee0 con 0x7fc988149b50 2024-03-29T10:39:37.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.486+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 >> v1:172.21.15.169:6789/0 conn(0x7fc98814fc70 legacy=0x7fc988152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.486+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 >> v1:172.21.15.151:6789/0 conn(0x7fc9880b2620 legacy=0x7fc9880b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 1 monclient: found mon.noname-b 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98814a590 con 0x7fc988149b50 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98c932640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98c932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc97c001670 con 0x7fc988149b50 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98559a640 10 monclient: dump: 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:37.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.487+0000 7fc98c932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.488+0000 7fc98559a640 1 -- 172.21.15.151:0/402308306 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc97c0017d0 con 0x7fc988149b50 2024-03-29T10:39:37.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.488+0000 7fc98559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.488+0000 7fc98c932640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:37.474 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.488+0000 7fc98c932640 4 set_mon_vals no callback set 2024-03-29T10:39:37.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.489+0000 7fc98c932640 10 monclient: shutdown 2024-03-29T10:39:37.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.489+0000 7fc98c932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.489+0000 7fc98c932640 1 -- 172.21.15.151:0/402308306 >> v1:172.21.15.151:6790/0 conn(0x7fc988149b50 legacy=0x7fc988143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.493+0000 7fc98c932640 1 -- 172.21.15.151:0/402308306 shutdown_connections 2024-03-29T10:39:37.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.493+0000 7fc98c932640 1 -- 172.21.15.151:0/402308306 >> 172.21.15.151:0/402308306 conn(0x7fc98806d180 msgr2=0x7fc988066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.494+0000 7fc98c932640 1 -- 172.21.15.151:0/402308306 shutdown_connections 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.496+0000 7fc98c932640 1 -- 172.21.15.151:0/402308306 wait complete. 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.497+0000 7fc98c932640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.497+0000 7fc98c932640 10 monclient: monmap: 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:37.481 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:37.482 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:37.482 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:37.482 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 librados: starting msgr at 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 librados: starting objecter 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 Processor -- start 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 -- start start 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 librados: setting wanted keys 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 1 librados: calling monclient init 2024-03-29T10:39:37.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.498+0000 7fc98c932640 10 monclient: init 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.499+0000 7fc98c932640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.499+0000 7fc98c932640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.499+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-b con 0x7fc9880b2620 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.499+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-c con 0x7fc988149b50 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.499+0000 7fc98c932640 10 monclient(hunting): picked mon.noname-a con 0x7fc98814fc70 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:37.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.500+0000 7fc98c932640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc988144a50 con 0x7fc98814fc70 2024-03-29T10:39:37.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.500+0000 7fc98c932640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9881db5a0 con 0x7fc9880b2620 2024-03-29T10:39:37.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.500+0000 7fc98c932640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98806fb80 con 0x7fc988149b50 2024-03-29T10:39:37.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.500+0000 7fc98c932640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.500+0000 7fc98c932640 10 monclient(hunting): authenticate will time out at 1108.669552s 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.504+0000 7fc985d9b640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fc988149b50 0x7fc988070720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:44227/0 (socket says 172.21.15.151:44227) 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.504+0000 7fc985d9b640 1 -- 172.21.15.151:0/961894968 learned_addr learned my addr 172.21.15.151:0/961894968 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 client.?.objecter ms_handle_connect 0x7fc988149b50 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620648554 0 0) 0x7fc98806fb80 con 0x7fc988149b50 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:39:37.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc968004eb0 con 0x7fc988149b50 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1825770293 0 0) 0x7fc968004eb0 con 0x7fc988149b50 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc968005300 con 0x7fc988149b50 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc980001710 con 0x7fc988149b50 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:37.491 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:37.492 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:37.492 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:37.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.505+0000 7fc976ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:37.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 client.?.objecter ms_handle_connect 0x7fc9880b2620 2024-03-29T10:39:37.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4191842921 0 0) 0x7fc968005300 con 0x7fc988149b50 2024-03-29T10:39:37.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 >> v1:172.21.15.169:6789/0 conn(0x7fc9880b2620 legacy=0x7fc9881dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 >> v1:172.21.15.151:6789/0 conn(0x7fc98814fc70 legacy=0x7fc988137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 1 monclient: found mon.c 2024-03-29T10:39:37.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98806fd10 con 0x7fc988149b50 2024-03-29T10:39:37.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc976ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:39:37.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 10 monclient: _renew_subs 2024-03-29T10:39:37.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9881dc1d0 con 0x7fc988149b50 2024-03-29T10:39:37.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:37.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 10 monclient: _renew_subs 2024-03-29T10:39:37.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9881478c0 con 0x7fc988149b50 2024-03-29T10:39:37.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.507+0000 7fc98c932640 1 librados: init done 2024-03-29T10:39:37.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc980001970 con 0x7fc988149b50 2024-03-29T10:39:37.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc980001710 con 0x7fc988149b50 2024-03-29T10:39:37.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:37.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc976ffd640 10 monclient: dump: 2024-03-29T10:39:37.506 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:37.506 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:37.506 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.506 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:37.507 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.511+0000 7fc977fff640 4 set_mon_vals no callback set 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.512+0000 7fc98c932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.512+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fc98001ac80 con 0x7fc988149b50 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc98c932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc98c932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c005080 con 0x7fc988149b50 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fc980084c00 con 0x7fc988149b50 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc976ffd640 10 client.4322.objecter ms_dispatch 0x7fc988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc976ffd640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc976ffd640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc976ffd640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T10:39:37.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.513+0000 7fc98659c640 10 monclient: get_auth_request con 0x7fc96806e060 auth_method 0 2024-03-29T10:39:37.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.514+0000 7fc976ffd640 10 client.4322.objecter ms_handle_connect 0x7fc96806e060 2024-03-29T10:39:37.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.520+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9800504e0 con 0x7fc988149b50 2024-03-29T10:39:37.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.521+0000 7fc976ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.521+0000 7fc976ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:37.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.852+0000 7fc98c932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.852+0000 7fc98c932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.852+0000 7fc98c932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:39:37.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.852+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc96c005330 con 0x7fc988149b50 2024-03-29T10:39:37.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.853+0000 7fc976ffd640 1 -- 172.21.15.151:0/961894968 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fc980052410 con 0x7fc988149b50 2024-03-29T10:39:37.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.853+0000 7fc976ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.853+0000 7fc976ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:37.840 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:37.840 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:37.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 10 librados: watch_flush enter 2024-03-29T10:39:37.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 10 librados: watch_flush exit 2024-03-29T10:39:37.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T10:39:37.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 10 client.4322.objecter successfully canceled tick 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 >> v1:172.21.15.151:6816/1549873000 conn(0x7fc96806e060 legacy=0x7fc968070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 10 monclient: shutdown 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 >> v1:172.21.15.151:6790/0 conn(0x7fc988149b50 legacy=0x7fc988070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 shutdown_connections 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.860+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 >> 172.21.15.151:0/961894968 conn(0x7fc98806d180 msgr2=0x7fc988066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.861+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 shutdown_connections 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.861+0000 7fc98c932640 1 -- 172.21.15.151:0/961894968 wait complete. 2024-03-29T10:39:37.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:37.861+0000 7fc98c932640 1 librados: shutdown 2024-03-29T10:39:37.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:44.668 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:44.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.836+0000 7f300d9a8640 10 monclient: get_monmap_and_config 2024-03-29T10:39:44.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.836+0000 7f300d9a8640 10 monclient: build_initial_monmap 2024-03-29T10:39:44.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.836+0000 7f300d9a8640 10 monclient: monmap: 2024-03-29T10:39:44.820 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:44.820 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:44.837698+0000 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:44.837698+0000 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:44.821 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.822 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.838+0000 7f300d9a8640 1 Processor -- start 2024-03-29T10:39:44.822 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.838+0000 7f300d9a8640 1 -- start start 2024-03-29T10:39:44.822 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.838+0000 7f300d9a8640 10 monclient: init 2024-03-29T10:39:44.823 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.839+0000 7f300d9a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:44.823 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.839+0000 7f300d9a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:44.823 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.839+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f30080b1f60 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:44.824 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.839+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f3008071eb0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.839+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f3008143d10 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.842+0000 7f300d9a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300806ef80 con 0x7f3008071eb0 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.842+0000 7f300d9a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300806f260 con 0x7f3008143d10 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.842+0000 7f300d9a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080b40c0 con 0x7f30080b1f60 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.842+0000 7f300d9a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:44.826 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.842+0000 7f300d9a8640 10 monclient(hunting): authenticate will time out at 1116.011711s 2024-03-29T10:39:44.827 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.843+0000 7f30077fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3008143d10 0x7f3008146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:33675/0 (socket says 172.21.15.151:33675) 2024-03-29T10:39:44.827 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.843+0000 7f30077fe640 1 -- 172.21.15.151:0/1594724285 learned_addr learned my addr 172.21.15.151:0/1594724285 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:44.828 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794524789 0 0) 0x7f300806f260 con 0x7f3008143d10 2024-03-29T10:39:44.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:44.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:44.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:44.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:39:44.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.845+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0003520 con 0x7f3008143d10 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103682742 0 0) 0x7f2ff0003520 con 0x7f3008143d10 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 >> v1:172.21.15.151:6790/0 conn(0x7f30080b1f60 legacy=0x7f3008143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 >> v1:172.21.15.151:6789/0 conn(0x7f3008071eb0 legacy=0x7f3008146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 monclient: found mon.noname-c 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3008155d70 con 0x7f3008143d10 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:44.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3000001a10 con 0x7f3008143d10 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.849+0000 7f3005ffb640 10 monclient: dump: 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:44.834 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.835 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:44.835 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:44.835 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:44.835 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f300d9a8640 5 monclient: authenticate success, global_id 4345 2024-03-29T10:39:44.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3000001ca0 con 0x7f3008143d10 2024-03-29T10:39:44.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f300d9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:44.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f3005ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:44.836 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f3005ffb640 1 -- 172.21.15.151:0/1594724285 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3000002030 con 0x7f3008143d10 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f300d9a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f300d9a8640 4 set_mon_vals no callback set 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.850+0000 7f3005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f3005ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f3005ffb640 10 monclient: dump: 2024-03-29T10:39:44.837 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.838 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f300d9a8640 10 monclient: shutdown 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f300d9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:44.839 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f300d9a8640 1 -- 172.21.15.151:0/1594724285 >> v1:172.21.15.169:6789/0 conn(0x7f3008143d10 legacy=0x7f3008146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:44.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.851+0000 7f300d9a8640 1 -- 172.21.15.151:0/1594724285 shutdown_connections 2024-03-29T10:39:44.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.852+0000 7f300d9a8640 1 -- 172.21.15.151:0/1594724285 >> 172.21.15.151:0/1594724285 conn(0x7f3008066a00 msgr2=0x7f3008066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:44.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.852+0000 7f300d9a8640 1 -- 172.21.15.151:0/1594724285 shutdown_connections 2024-03-29T10:39:44.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.852+0000 7f300d9a8640 1 -- 172.21.15.151:0/1594724285 wait complete. 2024-03-29T10:39:44.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.853+0000 7f300d9a8640 10 monclient: build_initial_monmap 2024-03-29T10:39:44.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.853+0000 7f300d9a8640 10 monclient: monmap: 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:44.843 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.853+0000 7f300d9a8640 1 librados: starting msgr at 2024-03-29T10:39:44.844 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.853+0000 7f300d9a8640 1 librados: starting objecter 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.854+0000 7f300d9a8640 1 Processor -- start 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.854+0000 7f300d9a8640 1 -- start start 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.854+0000 7f300d9a8640 1 librados: setting wanted keys 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.854+0000 7f300d9a8640 1 librados: calling monclient init 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.854+0000 7f300d9a8640 10 monclient: init 2024-03-29T10:39:44.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.855+0000 7f300d9a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.855+0000 7f300d9a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f30080b1f60 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f3008143d10 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f30080aa870 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080ab390 con 0x7f30080aa870 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080abf30 con 0x7f3008143d10 2024-03-29T10:39:44.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080ac210 con 0x7f30080b1f60 2024-03-29T10:39:44.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:44.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.856+0000 7f300d9a8640 10 monclient(hunting): authenticate will time out at 1116.025633s 2024-03-29T10:39:44.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.858+0000 7f3006ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f30080b1f60 0x7f30080a9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:50959/0 (socket says 172.21.15.151:50959) 2024-03-29T10:39:44.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.858+0000 7f3006ffd640 1 -- 172.21.15.151:0/1864038252 learned_addr learned my addr 172.21.15.151:0/1864038252 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:44.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.861+0000 7f2feffff640 10 client.?.objecter ms_handle_connect 0x7f30080b1f60 2024-03-29T10:39:44.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.861+0000 7f2feffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:44.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.861+0000 7f2feffff640 10 client.?.objecter ms_handle_connect 0x7f3008143d10 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.861+0000 7f2feffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.861+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166333043 0 0) 0x7f30080abf30 con 0x7f3008143d10 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff4004eb0 con 0x7f3008143d10 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373992163 0 0) 0x7f30080ac210 con 0x7f30080b1f60 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:44.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30080abf30 con 0x7f30080b1f60 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 client.?.objecter ms_handle_connect 0x7f30080aa870 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 616443209 0 0) 0x7f2ff4004eb0 con 0x7f3008143d10 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30080ac210 con 0x7f3008143d10 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2ff8001510 con 0x7f3008143d10 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): dump: 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.855 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441156453 0 0) 0x7f30080ab390 con 0x7f30080aa870 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:44.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.862+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff4004eb0 con 0x7f30080aa870 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 798189246 0 0) 0x7f30080ac210 con 0x7f3008143d10 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 >> v1:172.21.15.151:6790/0 conn(0x7f30080b1f60 legacy=0x7f30080a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 >> v1:172.21.15.151:6789/0 conn(0x7f30080aa870 legacy=0x7f300813fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 monclient: found mon.b 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080aae70 con 0x7f3008143d10 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: _finish_auth 0 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f300d9a8640 5 monclient: authenticate success, global_id 4348 2024-03-29T10:39:44.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f300d9a8640 10 monclient: _renew_subs 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f300d9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3008153ca0 con 0x7f3008143d10 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ff8001aa0 con 0x7f3008143d10 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2ff8001d50 con 0x7f3008143d10 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:44.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f2feffff640 10 monclient: dump: 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:44.859 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.863+0000 7f3004ff9640 4 set_mon_vals no callback set 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.864+0000 7f300d9a8640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.864+0000 7f300d9a8640 10 monclient: _renew_subs 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.864+0000 7f300d9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.864+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30081403f0 con 0x7f3008143d10 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.864+0000 7f300d9a8640 1 librados: init done 2024-03-29T10:39:44.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.866+0000 7f300d9a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.867+0000 7f300d9a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.867+0000 7f300d9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.867+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30080aace0 con 0x7f3008143d10 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.868+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f2ff8019920 con 0x7f3008143d10 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.869+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f2ff8083820 con 0x7f3008143d10 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.869+0000 7f2feffff640 10 client.4348.objecter ms_dispatch 0x7f3008000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.869+0000 7f2feffff640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:44.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.869+0000 7f2feffff640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:44.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.869+0000 7f2feffff640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T10:39:44.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.870+0000 7f3006ffd640 10 monclient: get_auth_request con 0x7f2ff406de70 auth_method 0 2024-03-29T10:39:44.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.870+0000 7f2feffff640 10 client.4348.objecter ms_handle_connect 0x7f2ff406de70 2024-03-29T10:39:44.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.879+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ff804f100 con 0x7f3008143d10 2024-03-29T10:39:44.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.879+0000 7f2feffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:44.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:44.879+0000 7f2feffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:45.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.200+0000 7f300d9a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:45.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.200+0000 7f300d9a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:45.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.200+0000 7f300d9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:39:45.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.200+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30080ab000 con 0x7f3008143d10 2024-03-29T10:39:45.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.201+0000 7f2feffff640 1 -- 172.21.15.151:0/1864038252 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f2ff8051030 con 0x7f3008143d10 2024-03-29T10:39:45.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.201+0000 7f2feffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:45.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.201+0000 7f2feffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:45.186 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:45.186 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:45.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.204+0000 7f300d9a8640 10 librados: watch_flush enter 2024-03-29T10:39:45.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.208+0000 7f300d9a8640 10 librados: watch_flush exit 2024-03-29T10:39:45.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.208+0000 7f300d9a8640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T10:39:45.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.208+0000 7f300d9a8640 10 client.4348.objecter successfully canceled tick 2024-03-29T10:39:45.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.208+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 >> v1:172.21.15.151:6816/1549873000 conn(0x7f2ff406de70 legacy=0x7f2ff4070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:45.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 10 monclient: shutdown 2024-03-29T10:39:45.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:45.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 >> v1:172.21.15.169:6789/0 conn(0x7f3008143d10 legacy=0x7f3008153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:45.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 shutdown_connections 2024-03-29T10:39:45.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 >> 172.21.15.151:0/1864038252 conn(0x7f3008066a00 msgr2=0x7f30080b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:45.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 shutdown_connections 2024-03-29T10:39:45.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 -- 172.21.15.151:0/1864038252 wait complete. 2024-03-29T10:39:45.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:45.209+0000 7f300d9a8640 1 librados: shutdown 2024-03-29T10:39:45.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:52.018 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:52.173 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.189+0000 7f0baf982640 10 monclient: get_monmap_and_config 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.189+0000 7f0baf982640 10 monclient: build_initial_monmap 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.189+0000 7f0baf982640 10 monclient: monmap: 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:52.190504+0000 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:52.190504+0000 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:52.174 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:52.175 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.190+0000 7f0baf982640 1 Processor -- start 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.190+0000 7f0baf982640 1 -- start start 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.190+0000 7f0baf982640 10 monclient: init 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba80b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba814a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba80ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80aba70 con 0x7f0ba80b0770 2024-03-29T10:39:52.178 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80aecb0 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.179 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80af150 con 0x7f0ba814a700 2024-03-29T10:39:52.179 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:52.179 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.191+0000 7f0baf982640 10 monclient(hunting): authenticate will time out at 1123.361116s 2024-03-29T10:39:52.180 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.195+0000 7f0bad71e640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0ba80b0770 0x7f0ba8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59953/0 (socket says 172.21.15.151:59953) 2024-03-29T10:39:52.180 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.195+0000 7f0bad71e640 1 -- 172.21.15.151:0/172059766 learned_addr learned my addr 172.21.15.151:0/172059766 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:52.180 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668355361 0 0) 0x7f0ba80aba70 con 0x7f0ba80b0770 2024-03-29T10:39:52.181 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:52.181 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:52.181 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:52.181 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:39:52.181 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.196+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94003520 con 0x7f0ba80b0770 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967488171 0 0) 0x7f0ba80aecb0 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80aba70 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263110637 0 0) 0x7f0b94003520 con 0x7f0ba80b0770 2024-03-29T10:39:52.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 >> v1:172.21.15.151:6790/0 conn(0x7f0ba814a700 legacy=0x7f0ba80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 >> v1:172.21.15.169:6789/0 conn(0x7f0ba80ac5a0 legacy=0x7f0ba8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 monclient: found mon.noname-a 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80ac9d0 con 0x7f0ba80b0770 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: _finish_auth 0 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0baf982640 5 monclient: authenticate success, global_id 4320 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0baf982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0ba0001a10 con 0x7f0ba80b0770 2024-03-29T10:39:52.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0b9ffff640 10 monclient: dump: 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.199+0000 7f0baf982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.200+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba0001b70 con 0x7f0ba80b0770 2024-03-29T10:39:52.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.200+0000 7f0b9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.200+0000 7f0baf982640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.200+0000 7f0baf982640 4 set_mon_vals no callback set 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0b9ffff640 1 -- 172.21.15.151:0/172059766 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0ba0001f40 con 0x7f0ba80b0770 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 10 monclient: shutdown 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 1 -- 172.21.15.151:0/172059766 >> v1:172.21.15.151:6789/0 conn(0x7f0ba80b0770 legacy=0x7f0ba8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0b9ffff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 1 -- 172.21.15.151:0/172059766 shutdown_connections 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 1 -- 172.21.15.151:0/172059766 >> 172.21.15.151:0/172059766 conn(0x7f0ba806b660 msgr2=0x7f0ba806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 1 -- 172.21.15.151:0/172059766 shutdown_connections 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.201+0000 7f0baf982640 1 -- 172.21.15.151:0/172059766 wait complete. 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.202+0000 7f0baf982640 10 monclient: build_initial_monmap 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.202+0000 7f0baf982640 10 monclient: monmap: 2024-03-29T10:39:52.190 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.202+0000 7f0baf982640 1 librados: starting msgr at 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.202+0000 7f0baf982640 1 librados: starting objecter 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.203+0000 7f0baf982640 1 Processor -- start 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.203+0000 7f0baf982640 1 -- start start 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.203+0000 7f0baf982640 1 librados: setting wanted keys 2024-03-29T10:39:52.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.203+0000 7f0baf982640 1 librados: calling monclient init 2024-03-29T10:39:52.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.203+0000 7f0baf982640 10 monclient: init 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba80ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba80b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba814a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8126260 con 0x7f0ba80b0770 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8128200 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8128470 con 0x7f0ba814a700 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:52.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.204+0000 7f0baf982640 10 monclient(hunting): authenticate will time out at 1123.373688s 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.210+0000 7f0bacf1d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0ba80b0770 0x7f0ba8126e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60261/0 (socket says 172.21.15.151:60261) 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.210+0000 7f0bacf1d640 1 -- 172.21.15.151:0/3742769703 learned_addr learned my addr 172.21.15.151:0/3742769703 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 client.?.objecter ms_handle_connect 0x7f0ba80b0770 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 client.?.objecter ms_handle_connect 0x7f0ba80ac5a0 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556564082 0 0) 0x7f0ba8126260 con 0x7f0ba80b0770 2024-03-29T10:39:52.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c004eb0 con 0x7f0ba80b0770 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633696009 0 0) 0x7f0ba8128200 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.211+0000 7f0b9dffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba8126260 con 0x7f0ba80ac5a0 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159534746 0 0) 0x7f0b8c004eb0 con 0x7f0ba80b0770 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba8128200 con 0x7f0ba80b0770 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0ba40017d0 con 0x7f0ba80b0770 2024-03-29T10:39:52.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 10 monclient(hunting): dump: 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3329836225 0 0) 0x7f0ba8128200 con 0x7f0ba80b0770 2024-03-29T10:39:52.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 >> v1:172.21.15.151:6790/0 conn(0x7f0ba814a700 legacy=0x7f0ba81e6000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 >> v1:172.21.15.169:6789/0 conn(0x7f0ba80ac5a0 legacy=0x7f0ba81266f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 1 monclient: found mon.a 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.212+0000 7f0b9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0b9dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8125e10 con 0x7f0ba80b0770 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0b9dffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0b9dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0b9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 10 monclient: _renew_subs 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba81df6f0 con 0x7f0ba80b0770 2024-03-29T10:39:52.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 10 monclient: _renew_subs 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba81deeb0 con 0x7f0ba80b0770 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.213+0000 7f0baf982640 1 librados: init done 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba4001930 con 0x7f0ba80b0770 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0ba4004a00 con 0x7f0ba80b0770 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9dffb640 10 monclient: dump: 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:52.203 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0baf982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0baf982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0baf982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba80ab560 con 0x7f0ba80b0770 2024-03-29T10:39:52.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.214+0000 7f0b9f7fe640 4 set_mon_vals no callback set 2024-03-29T10:39:52.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.216+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f0ba401a9d0 con 0x7f0ba80b0770 2024-03-29T10:39:52.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f0ba40849c0 con 0x7f0ba80b0770 2024-03-29T10:39:52.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0b9dffb640 10 client.4323.objecter ms_dispatch 0x7f0ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:52.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0b9dffb640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:52.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0b9dffb640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:52.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0b9dffb640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T10:39:52.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.217+0000 7f0bad71e640 10 monclient: get_auth_request con 0x7f0b8c06da80 auth_method 0 2024-03-29T10:39:52.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.218+0000 7f0b9dffb640 10 client.4323.objecter ms_handle_connect 0x7f0b8c06da80 2024-03-29T10:39:52.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.224+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba40500d0 con 0x7f0ba80b0770 2024-03-29T10:39:52.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.224+0000 7f0b9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:52.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.224+0000 7f0b9dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:52.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.542+0000 7f0baf982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:52.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.542+0000 7f0baf982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:52.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.542+0000 7f0baf982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:52.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.542+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ba8075f10 con 0x7f0ba80b0770 2024-03-29T10:39:52.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.543+0000 7f0b9dffb640 1 -- 172.21.15.151:0/3742769703 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f0ba40521d0 con 0x7f0ba80b0770 2024-03-29T10:39:52.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.543+0000 7f0b9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:52.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.543+0000 7f0b9dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:52.527 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:52.527 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:52.530 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 10 librados: watch_flush enter 2024-03-29T10:39:52.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 10 librados: watch_flush exit 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 10 client.4323.objecter successfully canceled tick 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 >> v1:172.21.15.151:6816/1549873000 conn(0x7f0b8c06da80 legacy=0x7f0b8c06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 10 monclient: shutdown 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 >> v1:172.21.15.151:6789/0 conn(0x7f0ba80b0770 legacy=0x7f0ba8126e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.547+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 shutdown_connections 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.548+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 >> 172.21.15.151:0/3742769703 conn(0x7f0ba806b660 msgr2=0x7f0ba80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.548+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 shutdown_connections 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.548+0000 7f0baf982640 1 -- 172.21.15.151:0/3742769703 wait complete. 2024-03-29T10:39:52.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:52.548+0000 7f0baf982640 1 librados: shutdown 2024-03-29T10:39:52.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:59.357 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:59.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.520+0000 7f01ad664640 10 monclient: get_monmap_and_config 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.520+0000 7f01ad664640 10 monclient: build_initial_monmap 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.520+0000 7f01ad664640 10 monclient: monmap: 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:39:59.521853+0000 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:39:59.521853+0000 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:39:59.505 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.525+0000 7f01ad664640 1 Processor -- start 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 1 -- start start 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 10 monclient: init 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-a con 0x7f01a81422b0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.527+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-b con 0x7f01a8142690 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-c con 0x7f01a80b17d0 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a8142c80 con 0x7f01a81422b0 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80aae40 con 0x7f01a80b17d0 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a813fbd0 con 0x7f01a8142690 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:59.513 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.528+0000 7f01ad664640 10 monclient(hunting): authenticate will time out at 1130.697565s 2024-03-29T10:39:59.514 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a67fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f01a8142690 0x7f01a80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:38149/0 (socket says 172.21.15.151:38149) 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a67fc640 1 -- 172.21.15.151:0/582954415 learned_addr learned my addr 172.21.15.151:0/582954415 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897734816 0 0) 0x7f01a813fbd0 con 0x7f01a8142690 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:59.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:39:59.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.531+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01a8142690 2024-03-29T10:39:59.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.532+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440704492 0 0) 0x7f0190003520 con 0x7f01a8142690 2024-03-29T10:39:59.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.532+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 >> v1:172.21.15.169:6789/0 conn(0x7f01a80b17d0 legacy=0x7f01a80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:59.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 >> v1:172.21.15.151:6789/0 conn(0x7f01a81422b0 legacy=0x7f01a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8151940 con 0x7f01a8142690 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0198001690 con 0x7f01a8142690 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.533+0000 7f01a5ffb640 10 monclient: dump: 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:59.519 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.535+0000 7f01ad664640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:39:59.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.535+0000 7f01ad664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:59.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01980018f0 con 0x7f01a8142690 2024-03-29T10:39:59.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 1 -- 172.21.15.151:0/582954415 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0198001cc0 con 0x7f01a8142690 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01a5ffb640 10 monclient: dump: 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:59.522 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01ad664640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.537+0000 7f01ad664640 4 set_mon_vals no callback set 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.538+0000 7f01ad664640 10 monclient: shutdown 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.538+0000 7f01ad664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:59.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.538+0000 7f01ad664640 1 -- 172.21.15.151:0/582954415 >> v1:172.21.15.151:6790/0 conn(0x7f01a8142690 legacy=0x7f01a80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:59.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.538+0000 7f01ad664640 1 -- 172.21.15.151:0/582954415 shutdown_connections 2024-03-29T10:39:59.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.538+0000 7f01ad664640 1 -- 172.21.15.151:0/582954415 >> 172.21.15.151:0/582954415 conn(0x7f01a8066a00 msgr2=0x7f01a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:59.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.539+0000 7f01ad664640 1 -- 172.21.15.151:0/582954415 shutdown_connections 2024-03-29T10:39:59.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.540+0000 7f01ad664640 1 -- 172.21.15.151:0/582954415 wait complete. 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.541+0000 7f01ad664640 10 monclient: build_initial_monmap 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.541+0000 7f01ad664640 10 monclient: monmap: 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:39:59.527 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.541+0000 7f01ad664640 1 librados: starting msgr at 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.541+0000 7f01ad664640 1 librados: starting objecter 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.541+0000 7f01ad664640 1 Processor -- start 2024-03-29T10:39:59.528 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.542+0000 7f01ad664640 1 -- start start 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.542+0000 7f01ad664640 1 librados: setting wanted keys 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.542+0000 7f01ad664640 1 librados: calling monclient init 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.542+0000 7f01ad664640 10 monclient: init 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-c con 0x7f01a80b17d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-a con 0x7f01a81422b0 addr v1:172.21.15.151:6789/0 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient(hunting): picked mon.noname-b con 0x7f01a8142690 addr v1:172.21.15.169:6789/0 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80acef0 con 0x7f01a81422b0 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80ad160 con 0x7f01a8142690 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80aba20 con 0x7f01a80b17d0 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:59.529 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.543+0000 7f01ad664640 10 monclient(hunting): authenticate will time out at 1130.713286s 2024-03-29T10:39:59.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.547+0000 7f01a6ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f01a80b17d0 0x7f01a81324a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:38269/0 (socket says 172.21.15.151:38269) 2024-03-29T10:39:59.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.547+0000 7f01a6ffd640 1 -- 172.21.15.151:0/2227928777 learned_addr learned my addr 172.21.15.151:0/2227928777 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:39:59.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 client.?.objecter ms_handle_connect 0x7f01a81422b0 2024-03-29T10:39:59.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:59.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063047579 0 0) 0x7f01a80acef0 con 0x7f01a81422b0 2024-03-29T10:39:59.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.548+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0194004eb0 con 0x7f01a81422b0 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707371641 0 0) 0x7f0194004eb0 con 0x7f01a81422b0 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a80acef0 con 0x7f01a81422b0 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0198001c00 con 0x7f01a81422b0 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.539 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.549+0000 7f018f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3953431043 0 0) 0x7f01a80acef0 con 0x7f01a81422b0 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 >> v1:172.21.15.151:6790/0 conn(0x7f01a80b17d0 legacy=0x7f01a81324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 >> v1:172.21.15.169:6789/0 conn(0x7f01a8142690 legacy=0x7f01a8130690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 1 monclient: found mon.a 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:59.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80abcb0 con 0x7f01a81422b0 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.551+0000 7f018f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0198001d60 con 0x7f01a81422b0 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0198004d30 con 0x7f01a81422b0 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f018f7fe640 10 monclient: dump: 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:39:59.541 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.552+0000 7f01a57fa640 4 set_mon_vals no callback set 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 10 monclient: _renew_subs 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a81e2c90 con 0x7f01a81422b0 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 10 monclient: _renew_subs 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a81e3000 con 0x7f01a81422b0 2024-03-29T10:39:59.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.553+0000 7f01ad664640 1 librados: init done 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.555+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f0198004e90 con 0x7f01a81422b0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f0198084c20 con 0x7f01a81422b0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f018f7fe640 10 client.4326.objecter ms_dispatch 0x7f01a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f018f7fe640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f018f7fe640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f018f7fe640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.556+0000 7f01a6ffd640 10 monclient: get_auth_request con 0x7f019406dc90 auth_method 0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.557+0000 7f01ad664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.557+0000 7f01ad664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.557+0000 7f01ad664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.557+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a812fbf0 con 0x7f01a81422b0 2024-03-29T10:39:59.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.559+0000 7f018f7fe640 10 client.4326.objecter ms_handle_connect 0x7f019406dc90 2024-03-29T10:39:59.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.566+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0198050500 con 0x7f01a81422b0 2024-03-29T10:39:59.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.566+0000 7f018f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:59.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.566+0000 7f018f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:59.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.887+0000 7f01ad664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:59.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.887+0000 7f01ad664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:59.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.887+0000 7f01ad664640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:39:59.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.887+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01a80670f0 con 0x7f01a81422b0 2024-03-29T10:39:59.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.888+0000 7f018f7fe640 1 -- 172.21.15.151:0/2227928777 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f0198052430 con 0x7f01a81422b0 2024-03-29T10:39:59.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.888+0000 7f018f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:59.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.888+0000 7f018f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:59.873 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:39:59.873 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 10 librados: watch_flush enter 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 10 librados: watch_flush exit 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 10 client.4326.objecter successfully canceled tick 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 >> v1:172.21.15.151:6816/1549873000 conn(0x7f019406dc90 legacy=0x7f0194070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 10 monclient: shutdown 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:59.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.892+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 >> v1:172.21.15.151:6789/0 conn(0x7f01a81422b0 legacy=0x7f01a812ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:59.879 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.895+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 shutdown_connections 2024-03-29T10:39:59.879 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.895+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 >> 172.21.15.151:0/2227928777 conn(0x7f01a8066a00 msgr2=0x7f01a813c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:59.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.898+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 shutdown_connections 2024-03-29T10:39:59.884 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.901+0000 7f01ad664640 1 -- 172.21.15.151:0/2227928777 wait complete. 2024-03-29T10:39:59.885 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:39:59.901+0000 7f01ad664640 1 librados: shutdown 2024-03-29T10:39:59.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:06.709 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:06.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.862+0000 7efe5a6d3640 10 monclient: get_monmap_and_config 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.862+0000 7efe5a6d3640 10 monclient: build_initial_monmap 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.862+0000 7efe5a6d3640 10 monclient: monmap: 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:06.863813+0000 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:06.863813+0000 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:06.847 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.863+0000 7efe5a6d3640 1 Processor -- start 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.863+0000 7efe5a6d3640 1 -- start start 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.863+0000 7efe5a6d3640 10 monclient: init 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-b con 0x7efe540b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:06.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-c con 0x7efe5414a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-a con 0x7efe540ac5a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe540aba70 con 0x7efe540ac5a0 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe540aecb0 con 0x7efe5414a700 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe540af150 con 0x7efe540b0770 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe5a6d3640 10 monclient(hunting): authenticate will time out at 1138.034371s 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe53fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7efe540b0770 0x7efe54149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:36222/0 (socket says 172.21.15.151:36222) 2024-03-29T10:40:06.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.864+0000 7efe53fff640 1 -- 172.21.15.151:0/398727435 learned_addr learned my addr 172.21.15.151:0/398727435 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:06.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084802365 0 0) 0x7efe540af150 con 0x7efe540b0770 2024-03-29T10:40:06.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:06.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:06.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:06.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003520 con 0x7efe540b0770 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367485334 0 0) 0x7efe540aba70 con 0x7efe540ac5a0 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe540af150 con 0x7efe540ac5a0 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8109580 0 0) 0x7efe3c003520 con 0x7efe540b0770 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.865+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 >> v1:172.21.15.169:6789/0 conn(0x7efe5414a700 legacy=0x7efe540abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 >> v1:172.21.15.151:6789/0 conn(0x7efe540ac5a0 legacy=0x7efe54151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 1 monclient: found mon.noname-b 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:06.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe540ac9d0 con 0x7efe540b0770 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7efe44001690 con 0x7efe540b0770 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 10 monclient: dump: 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:06.855 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe5a6d3640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe5a6d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.867+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe440017f0 con 0x7efe540b0770 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe52ffd640 1 -- 172.21.15.151:0/398727435 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7efe44001c40 con 0x7efe540b0770 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe52ffd640 10 monclient: dump: 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:06.856 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe5a6d3640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.868+0000 7efe5a6d3640 4 set_mon_vals no callback set 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 10 monclient: shutdown 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 1 -- 172.21.15.151:0/398727435 >> v1:172.21.15.151:6790/0 conn(0x7efe540b0770 legacy=0x7efe54149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:06.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 1 -- 172.21.15.151:0/398727435 shutdown_connections 2024-03-29T10:40:06.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 1 -- 172.21.15.151:0/398727435 >> 172.21.15.151:0/398727435 conn(0x7efe5406b660 msgr2=0x7efe5406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:06.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 1 -- 172.21.15.151:0/398727435 shutdown_connections 2024-03-29T10:40:06.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.869+0000 7efe5a6d3640 1 -- 172.21.15.151:0/398727435 wait complete. 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.870+0000 7efe5a6d3640 10 monclient: build_initial_monmap 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.870+0000 7efe5a6d3640 10 monclient: monmap: 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:06.865 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.870+0000 7efe5a6d3640 1 librados: starting msgr at 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.870+0000 7efe5a6d3640 1 librados: starting objecter 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.870+0000 7efe5a6d3640 1 Processor -- start 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 1 -- start start 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 1 librados: setting wanted keys 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 1 librados: calling monclient init 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 10 monclient: init 2024-03-29T10:40:06.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-b con 0x7efe540ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.871+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-c con 0x7efe540b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 10 monclient(hunting): picked mon.noname-a con 0x7efe5414a700 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe54127fc0 con 0x7efe5414a700 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe54125c40 con 0x7efe540ac5a0 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe541239d0 con 0x7efe540b0770 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe5a6d3640 10 monclient(hunting): authenticate will time out at 1138.041775s 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe58c70640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7efe5414a700 0x7efe541e5fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42110/0 (socket says 172.21.15.151:42110) 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe58c70640 1 -- 172.21.15.151:0/1411099354 learned_addr learned my addr 172.21.15.151:0/1411099354 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe50ff9640 10 client.?.objecter ms_handle_connect 0x7efe5414a700 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe50ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe50ff9640 10 client.?.objecter ms_handle_connect 0x7efe540b0770 2024-03-29T10:40:06.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe50ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.872+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584671110 0 0) 0x7efe54127fc0 con 0x7efe5414a700 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe40004eb0 con 0x7efe5414a700 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320242213 0 0) 0x7efe541239d0 con 0x7efe540b0770 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe54127fc0 con 0x7efe540b0770 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 134172683 0 0) 0x7efe40004eb0 con 0x7efe5414a700 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe541239d0 con 0x7efe5414a700 2024-03-29T10:40:06.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7efe4c001dd0 con 0x7efe5414a700 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:06.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.873+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1318847003 0 0) 0x7efe541239d0 con 0x7efe5414a700 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 >> v1:172.21.15.151:6790/0 conn(0x7efe540b0770 legacy=0x7efe54126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 >> v1:172.21.15.169:6789/0 conn(0x7efe540ac5a0 legacy=0x7efe541dba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 1 monclient: found mon.a 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54123b60 con 0x7efe5414a700 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 10 monclient: _renew_subs 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:06.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe541dc180 con 0x7efe5414a700 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 10 monclient: _renew_subs 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe54126f80 con 0x7efe5414a700 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.874+0000 7efe5a6d3640 1 librados: init done 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.875+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe4c001f30 con 0x7efe5414a700 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.875+0000 7efe50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.875+0000 7efe5a6d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.875+0000 7efe527fc640 4 set_mon_vals no callback set 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7efe4c003c20 con 0x7efe5414a700 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe50ff9640 10 monclient: dump: 2024-03-29T10:40:06.871 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe5a6d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe5a6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.876+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe540ab560 con 0x7efe5414a700 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.878+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7efe4c003d80 con 0x7efe5414a700 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.878+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7efe4c084db0 con 0x7efe5414a700 2024-03-29T10:40:06.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.878+0000 7efe50ff9640 10 client.4332.objecter ms_dispatch 0x7efe54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.878+0000 7efe50ff9640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.878+0000 7efe50ff9640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.879+0000 7efe53fff640 10 monclient: get_auth_request con 0x7efe4006da30 auth_method 0 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.879+0000 7efe50ff9640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.879+0000 7efe50ff9640 10 client.4332.objecter ms_handle_connect 0x7efe4006da30 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.886+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe4c050690 con 0x7efe5414a700 2024-03-29T10:40:06.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.886+0000 7efe50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:06.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:06.886+0000 7efe50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:07.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.204+0000 7efe5a6d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:07.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.204+0000 7efe5a6d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:07.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.204+0000 7efe5a6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:07.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.204+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe54075f10 con 0x7efe5414a700 2024-03-29T10:40:07.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.205+0000 7efe50ff9640 1 -- 172.21.15.151:0/1411099354 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7efe4c0525c0 con 0x7efe5414a700 2024-03-29T10:40:07.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.205+0000 7efe50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:07.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.205+0000 7efe50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:07.189 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:07.189 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.208+0000 7efe5a6d3640 10 librados: watch_flush enter 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.208+0000 7efe5a6d3640 10 librados: watch_flush exit 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.208+0000 7efe5a6d3640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.208+0000 7efe5a6d3640 10 client.4332.objecter successfully canceled tick 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 >> v1:172.21.15.151:6816/1549873000 conn(0x7efe4006da30 legacy=0x7efe4006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 10 monclient: shutdown 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:07.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 >> v1:172.21.15.151:6789/0 conn(0x7efe5414a700 legacy=0x7efe541e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:07.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 shutdown_connections 2024-03-29T10:40:07.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 >> 172.21.15.151:0/1411099354 conn(0x7efe5406b660 msgr2=0x7efe540713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:07.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 shutdown_connections 2024-03-29T10:40:07.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.209+0000 7efe5a6d3640 1 -- 172.21.15.151:0/1411099354 wait complete. 2024-03-29T10:40:07.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:07.210+0000 7efe5a6d3640 1 librados: shutdown 2024-03-29T10:40:07.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:14.017 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:14.175 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.191+0000 7f7c89c8f640 10 monclient: get_monmap_and_config 2024-03-29T10:40:14.175 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.191+0000 7f7c89c8f640 10 monclient: build_initial_monmap 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.191+0000 7f7c89c8f640 10 monclient: monmap: 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:14.192565+0000 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:14.192565+0000 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:14.176 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.177 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:14.177 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:14.177 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:14.177 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.182 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.192+0000 7f7c89c8f640 1 Processor -- start 2024-03-29T10:40:14.182 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.193+0000 7f7c89c8f640 1 -- start start 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.193+0000 7f7c89c8f640 10 monclient: init 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c840b0770 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c8414a700 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c840ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:14.183 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840aba70 con 0x7f7c8414a700 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c83fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f7c840ac5a0 0x7f7c84151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45166/0 (socket says 172.21.15.151:45166) 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c83fff640 1 -- 172.21.15.151:0/4170663844 learned_addr learned my addr 172.21.15.151:0/4170663844 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840aecb0 con 0x7f7c840b0770 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840af150 con 0x7f7c840ac5a0 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.194+0000 7f7c89c8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:14.184 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c89c8f640 10 monclient(hunting): authenticate will time out at 1145.364563s 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239959745 0 0) 0x7f7c840af150 con 0x7f7c840ac5a0 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:40:14.185 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c6c003520 con 0x7f7c840ac5a0 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346905689 0 0) 0x7f7c840aba70 con 0x7f7c8414a700 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:40:14.186 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c840af150 con 0x7f7c8414a700 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872170287 0 0) 0x7f7c840aecb0 con 0x7f7c840b0770 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.195+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c840aba70 con 0x7f7c840b0770 2024-03-29T10:40:14.187 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2749054567 0 0) 0x7f7c6c003520 con 0x7f7c840ac5a0 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 >> v1:172.21.15.169:6789/0 conn(0x7f7c840b0770 legacy=0x7f7c84149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 >> v1:172.21.15.151:6789/0 conn(0x7f7c8414a700 legacy=0x7f7c840abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c840ac9d0 con 0x7f7c840ac5a0 2024-03-29T10:40:14.188 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c7c001a10 con 0x7f7c840ac5a0 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c89c8f640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c89c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:14.189 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: dump: 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:14.190 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c89c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c7c001c30 con 0x7f7c840ac5a0 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.196+0000 7f7c827fc640 1 -- 172.21.15.151:0/4170663844 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c7c001fc0 con 0x7f7c840ac5a0 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.197+0000 7f7c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.197+0000 7f7c827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:14.191 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.197+0000 7f7c827fc640 10 monclient: dump: 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:14.192 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.197+0000 7f7c89c8f640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.197+0000 7f7c89c8f640 4 set_mon_vals no callback set 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 10 monclient: shutdown 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 >> v1:172.21.15.151:6790/0 conn(0x7f7c840ac5a0 legacy=0x7f7c84151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 shutdown_connections 2024-03-29T10:40:14.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 >> 172.21.15.151:0/4170663844 conn(0x7f7c8406b660 msgr2=0x7f7c8406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:14.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 shutdown_connections 2024-03-29T10:40:14.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.198+0000 7f7c89c8f640 1 -- 172.21.15.151:0/4170663844 wait complete. 2024-03-29T10:40:14.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.199+0000 7f7c89c8f640 10 monclient: build_initial_monmap 2024-03-29T10:40:14.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.199+0000 7f7c89c8f640 10 monclient: monmap: 2024-03-29T10:40:14.203 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.200+0000 7f7c89c8f640 1 librados: starting msgr at 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.200+0000 7f7c89c8f640 1 librados: starting objecter 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.200+0000 7f7c89c8f640 1 Processor -- start 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.201+0000 7f7c89c8f640 1 -- start start 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.201+0000 7f7c89c8f640 1 librados: setting wanted keys 2024-03-29T10:40:14.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.201+0000 7f7c89c8f640 1 librados: calling monclient init 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.201+0000 7f7c89c8f640 10 monclient: init 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.202+0000 7f7c89c8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.202+0000 7f7c89c8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c840ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c840b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c89c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c8414a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c837fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f7c840ac5a0 0x7f7c841df410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45370/0 (socket says 172.21.15.151:45370) 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c837fe640 1 -- 172.21.15.151:0/121853755 learned_addr learned my addr 172.21.15.151:0/121853755 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c841dfb20 con 0x7f7c840b0770 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c6bfff640 10 client.?.objecter ms_handle_connect 0x7f7c840ac5a0 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c6bfff640 10 client.?.objecter ms_handle_connect 0x7f7c8414a700 2024-03-29T10:40:14.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.203+0000 7f7c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c841dfcb0 con 0x7f7c8414a700 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 client.?.objecter ms_handle_connect 0x7f7c840b0770 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c841dfe40 con 0x7f7c840ac5a0 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c89c8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c89c8f640 10 monclient(hunting): authenticate will time out at 1145.373746s 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315331778 0 0) 0x7f7c841dfcb0 con 0x7f7c8414a700 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c70004eb0 con 0x7f7c8414a700 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.204+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352601132 0 0) 0x7f7c841dfe40 con 0x7f7c840ac5a0 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c841dfcb0 con 0x7f7c840ac5a0 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.205+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935348796 0 0) 0x7f7c841dfb20 con 0x7f7c840b0770 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:14.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c841dfe40 con 0x7f7c840b0770 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3438021712 0 0) 0x7f7c70004eb0 con 0x7f7c8414a700 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c841dfb20 con 0x7f7c8414a700 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768058402 0 0) 0x7f7c841dfcb0 con 0x7f7c840ac5a0 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c70004eb0 con 0x7f7c840ac5a0 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c7c001950 con 0x7f7c8414a700 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.208 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c74001870 con 0x7f7c840ac5a0 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.206+0000 7f7c6bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.209 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1075867544 0 0) 0x7f7c841dfb20 con 0x7f7c8414a700 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 >> v1:172.21.15.151:6790/0 conn(0x7f7c840ac5a0 legacy=0x7f7c841df410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 >> v1:172.21.15.151:6789/0 conn(0x7f7c840b0770 legacy=0x7f7c84126930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 monclient: found mon.b 2024-03-29T10:40:14.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c841453c0 con 0x7f7c8414a700 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: _finish_auth 0 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 10 monclient: _renew_subs 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c84145610 con 0x7f7c8414a700 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 10 monclient: _renew_subs 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c841e78f0 con 0x7f7c8414a700 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c89c8f640 1 librados: init done 2024-03-29T10:40:14.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c7c001ab0 con 0x7f7c8414a700 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c7c004fb0 con 0x7f7c8414a700 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.207+0000 7f7c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.208+0000 7f7c6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.208+0000 7f7c6bfff640 10 monclient: dump: 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:14.212 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.209+0000 7f7c817fa640 4 set_mon_vals no callback set 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.210+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f7c7c005110 con 0x7f7c8414a700 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.210+0000 7f7c89c8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c89c8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c89c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c64005080 con 0x7f7c8414a700 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c837fe640 10 monclient: get_auth_request con 0x7f7c7006e040 auth_method 0 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f7c7c084d70 con 0x7f7c8414a700 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c6bfff640 10 client.4360.objecter ms_dispatch 0x7f7c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:14.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c6bfff640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.211+0000 7f7c6bfff640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.212+0000 7f7c6bfff640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.212+0000 7f7c6bfff640 10 client.4360.objecter ms_handle_connect 0x7f7c7006e040 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.219+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c7c050480 con 0x7f7c8414a700 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.219+0000 7f7c6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:14.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.219+0000 7f7c6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:14.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.604+0000 7f7c89c8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:14.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.605+0000 7f7c89c8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:14.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.605+0000 7f7c89c8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:14.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.605+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c64005330 con 0x7f7c8414a700 2024-03-29T10:40:14.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.606+0000 7f7c6bfff640 1 -- 172.21.15.151:0/121853755 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f7c7c052580 con 0x7f7c8414a700 2024-03-29T10:40:14.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.606+0000 7f7c6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:14.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.606+0000 7f7c6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:14.590 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:14.590 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:14.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 10 librados: watch_flush enter 2024-03-29T10:40:14.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 10 librados: watch_flush exit 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 10 client.4360.objecter successfully canceled tick 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 >> v1:172.21.15.151:6816/1549873000 conn(0x7f7c7006e040 legacy=0x7f7c70070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 10 monclient: shutdown 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 >> v1:172.21.15.169:6789/0 conn(0x7f7c8414a700 legacy=0x7f7c84127040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 shutdown_connections 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.609+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 >> 172.21.15.151:0/121853755 conn(0x7f7c8406b660 msgr2=0x7f7c8406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.610+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 shutdown_connections 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.610+0000 7f7c89c8f640 1 -- 172.21.15.151:0/121853755 wait complete. 2024-03-29T10:40:14.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:14.610+0000 7f7c89c8f640 1 librados: shutdown 2024-03-29T10:40:14.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:21.417 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.583+0000 7fe6797cb640 10 monclient: get_monmap_and_config 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.583+0000 7fe6797cb640 10 monclient: build_initial_monmap 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.583+0000 7fe6797cb640 10 monclient: monmap: 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:21.584311+0000 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:21.584311+0000 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:21.567 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.568 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:21.568 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:21.568 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:21.568 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.584+0000 7fe6797cb640 1 Processor -- start 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.584+0000 7fe6797cb640 1 -- start start 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.584+0000 7fe6797cb640 10 monclient: init 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-a con 0x7fe6741457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe67413f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6740a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740aaef0 con 0x7fe6741457f0 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674145bd0 con 0x7fe6740a9d70 2024-03-29T10:40:21.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674145eb0 con 0x7fe67413f650 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6797cb640 10 monclient(hunting): authenticate will time out at 1152.754669s 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6727fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe67413f650 0x7fe6740b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:57262/0 (socket says 172.21.15.151:57262) 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe6727fc640 1 -- 172.21.15.151:0/4007315008 learned_addr learned my addr 172.21.15.151:0/4007315008 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729158867 0 0) 0x7fe6740aaef0 con 0x7fe6741457f0 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.585+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660003520 con 0x7fe6741457f0 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903992324 0 0) 0x7fe674145eb0 con 0x7fe67413f650 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:21.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6740aaef0 con 0x7fe67413f650 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573049595 0 0) 0x7fe660003520 con 0x7fe6741457f0 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 >> v1:172.21.15.151:6790/0 conn(0x7fe67413f650 legacy=0x7fe6740b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 >> v1:172.21.15.169:6789/0 conn(0x7fe6740a9d70 legacy=0x7fe67414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 monclient: found mon.noname-a 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674146340 con 0x7fe6741457f0 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:21.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe66c001610 con 0x7fe6741457f0 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: dump: 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe6797cb640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:40:21.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe6797cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe66c001770 con 0x7fe6741457f0 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe671ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe6797cb640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.586+0000 7fe6797cb640 4 set_mon_vals no callback set 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe671ffb640 1 -- 172.21.15.151:0/4007315008 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe66c001bc0 con 0x7fe6741457f0 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe671ffb640 10 monclient: dump: 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.576 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe6797cb640 10 monclient: shutdown 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe6797cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.587+0000 7fe6797cb640 1 -- 172.21.15.151:0/4007315008 >> v1:172.21.15.151:6789/0 conn(0x7fe6741457f0 legacy=0x7fe67413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.588+0000 7fe6797cb640 1 -- 172.21.15.151:0/4007315008 shutdown_connections 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.588+0000 7fe6797cb640 1 -- 172.21.15.151:0/4007315008 >> 172.21.15.151:0/4007315008 conn(0x7fe674066a00 msgr2=0x7fe674066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.588+0000 7fe6797cb640 1 -- 172.21.15.151:0/4007315008 shutdown_connections 2024-03-29T10:40:21.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.588+0000 7fe6797cb640 1 -- 172.21.15.151:0/4007315008 wait complete. 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.589+0000 7fe6797cb640 10 monclient: build_initial_monmap 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.589+0000 7fe6797cb640 10 monclient: monmap: 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:21.581 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.589+0000 7fe6797cb640 1 librados: starting msgr at 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.589+0000 7fe6797cb640 1 librados: starting objecter 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 1 Processor -- start 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 1 -- start start 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 1 librados: setting wanted keys 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 1 librados: calling monclient init 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 10 monclient: init 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:21.582 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.590+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6740a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-a con 0x7fe67413f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6741457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe672ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe6740a9d70 0x7fe6741405a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:55180/0 (socket says 172.21.15.151:55180) 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe672ffd640 1 -- 172.21.15.151:0/3676469623 learned_addr learned my addr 172.21.15.151:0/3676469623 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6727fc640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe67413f650 0x7fe674133940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:36618/0 (socket says 172.21.15.151:36618) 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674140110 con 0x7fe67413f650 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674135830 con 0x7fe6740a9d70 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe674134760 con 0x7fe6741457f0 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe6797cb640 10 monclient(hunting): authenticate will time out at 1152.760808s 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter ms_handle_connect 0x7fe67413f650 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter ms_handle_connect 0x7fe6741457f0 2024-03-29T10:40:21.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter ms_handle_connect 0x7fe6740a9d70 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655419944 0 0) 0x7fe674135830 con 0x7fe6740a9d70 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe654004eb0 con 0x7fe6740a9d70 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075634048 0 0) 0x7fe674134760 con 0x7fe6741457f0 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.591+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674135830 con 0x7fe6741457f0 2024-03-29T10:40:21.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488626160 0 0) 0x7fe674140110 con 0x7fe67413f650 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674134760 con 0x7fe67413f650 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1956250433 0 0) 0x7fe674134760 con 0x7fe67413f650 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe674140110 con 0x7fe67413f650 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe664001cb0 con 0x7fe67413f650 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:21.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.586 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022137884 0 0) 0x7fe674135830 con 0x7fe6741457f0 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe674134760 con 0x7fe6741457f0 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe65c001790 con 0x7fe6741457f0 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.587 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991774292 0 0) 0x7fe654004eb0 con 0x7fe6740a9d70 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe674135830 con 0x7fe6740a9d70 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe66c001d80 con 0x7fe6740a9d70 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.588 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1678309843 0 0) 0x7fe674140110 con 0x7fe67413f650 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 >> v1:172.21.15.151:6790/0 conn(0x7fe6741457f0 legacy=0x7fe674134050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 >> v1:172.21.15.169:6789/0 conn(0x7fe6740a9d70 legacy=0x7fe6741405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 monclient: found mon.a 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:21.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674140cb0 con 0x7fe67413f650 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe66b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 10 monclient: _renew_subs 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674135b10 con 0x7fe67413f650 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:21.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 10 monclient: _renew_subs 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe674134a70 con 0x7fe67413f650 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.592+0000 7fe6797cb640 1 librados: init done 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6640020c0 con 0x7fe67413f650 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe664004d00 con 0x7fe67413f650 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:21.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe66b7fe640 10 monclient: dump: 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.593+0000 7fe6717fa640 4 set_mon_vals no callback set 2024-03-29T10:40:21.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.594+0000 7fe6797cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.594+0000 7fe6797cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.594+0000 7fe6797cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.594+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe658005080 con 0x7fe67413f650 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe66401acd0 con 0x7fe67413f650 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe664084c30 con 0x7fe67413f650 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 10 client.4344.objecter ms_dispatch 0x7fe674000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe672ffd640 10 monclient: get_auth_request con 0x7fe65406e0a0 auth_method 0 2024-03-29T10:40:21.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.595+0000 7fe66b7fe640 10 client.4344.objecter ms_handle_connect 0x7fe65406e0a0 2024-03-29T10:40:21.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.607+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe664050510 con 0x7fe67413f650 2024-03-29T10:40:21.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.607+0000 7fe66b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:21.594 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.607+0000 7fe66b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:21.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.934+0000 7fe6797cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.935+0000 7fe6797cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.935+0000 7fe6797cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.935+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe658005330 con 0x7fe67413f650 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.936+0000 7fe66b7fe640 1 -- 172.21.15.151:0/3676469623 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe664052440 con 0x7fe67413f650 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.936+0000 7fe66b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:21.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.936+0000 7fe66b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:21.920 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:21.920 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 10 librados: watch_flush enter 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 10 librados: watch_flush exit 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 10 client.4344.objecter successfully canceled tick 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe65406e0a0 legacy=0x7fe654070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 10 monclient: shutdown 2024-03-29T10:40:21.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:21.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.939+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 >> v1:172.21.15.151:6789/0 conn(0x7fe67413f650 legacy=0x7fe674133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:21.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.940+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 shutdown_connections 2024-03-29T10:40:21.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.940+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 >> 172.21.15.151:0/3676469623 conn(0x7fe674066a00 msgr2=0x7fe6740b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:21.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.940+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 shutdown_connections 2024-03-29T10:40:21.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.940+0000 7fe6797cb640 1 -- 172.21.15.151:0/3676469623 wait complete. 2024-03-29T10:40:21.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:21.940+0000 7fe6797cb640 1 librados: shutdown 2024-03-29T10:40:21.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:28.746 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.912+0000 7f1fb35c8640 10 monclient: get_monmap_and_config 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.912+0000 7f1fb35c8640 10 monclient: build_initial_monmap 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.912+0000 7f1fb35c8640 10 monclient: monmap: 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:28.913283+0000 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:28.913283+0000 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.896 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:28.897 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:28.897 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:28.897 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.897 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.914+0000 7f1fb35c8640 1 Processor -- start 2024-03-29T10:40:28.898 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.914+0000 7f1fb35c8640 1 -- start start 2024-03-29T10:40:28.898 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.914+0000 7f1fb35c8640 10 monclient: init 2024-03-29T10:40:28.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:28.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:28.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1fac0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1fac14a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1fac0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac0aba70 con 0x7f1fac0b0770 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac0aecb0 con 0x7f1fac0ac5a0 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac0af150 con 0x7f1fac14a700 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb35c8640 10 monclient(hunting): authenticate will time out at 1160.085241s 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.915+0000 7f1fb0b63640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f1fac14a700 0x7f1fac0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60290/0 (socket says 172.21.15.151:60290) 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fb0b63640 1 -- 172.21.15.151:0/2966431316 learned_addr learned my addr 172.21.15.151:0/2966431316 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828505680 0 0) 0x7f1fac0af150 con 0x7f1fac14a700 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:28.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:28.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:40:28.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f98003520 con 0x7f1fac14a700 2024-03-29T10:40:28.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570167141 0 0) 0x7f1f98003520 con 0x7f1fac14a700 2024-03-29T10:40:28.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.916+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 >> v1:172.21.15.169:6789/0 conn(0x7f1fac0ac5a0 legacy=0x7f1fac151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:28.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 >> v1:172.21.15.151:6789/0 conn(0x7f1fac0b0770 legacy=0x7f1fac149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:28.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:28.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 1 monclient: found mon.noname-b 2024-03-29T10:40:28.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:28.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0ac9d0 con 0x7f1fac14a700 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1f9c001690 con 0x7f1fac14a700 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.919+0000 7f1fa3fff640 10 monclient: dump: 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:28.905 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fb35c8640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fb35c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f9c001920 con 0x7f1fac14a700 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 1 -- 172.21.15.151:0/2966431316 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1f9c001cb0 con 0x7f1fac14a700 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fa3fff640 10 monclient: dump: 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.906 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fb35c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.920+0000 7f1fb35c8640 4 set_mon_vals no callback set 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 10 monclient: shutdown 2024-03-29T10:40:28.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:28.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 1 -- 172.21.15.151:0/2966431316 >> v1:172.21.15.151:6790/0 conn(0x7f1fac14a700 legacy=0x7f1fac0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:28.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 1 -- 172.21.15.151:0/2966431316 shutdown_connections 2024-03-29T10:40:28.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 1 -- 172.21.15.151:0/2966431316 >> 172.21.15.151:0/2966431316 conn(0x7f1fac06b660 msgr2=0x7f1fac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:28.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.922+0000 7f1fb35c8640 1 -- 172.21.15.151:0/2966431316 shutdown_connections 2024-03-29T10:40:28.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.923+0000 7f1fb35c8640 1 -- 172.21.15.151:0/2966431316 wait complete. 2024-03-29T10:40:28.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.924+0000 7f1fb35c8640 10 monclient: build_initial_monmap 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.924+0000 7f1fb35c8640 10 monclient: monmap: 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.925+0000 7f1fb35c8640 1 librados: starting msgr at 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.925+0000 7f1fb35c8640 1 librados: starting objecter 2024-03-29T10:40:28.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.925+0000 7f1fb35c8640 1 Processor -- start 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.926+0000 7f1fb35c8640 1 -- start start 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.926+0000 7f1fb35c8640 1 librados: setting wanted keys 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.926+0000 7f1fb35c8640 1 librados: calling monclient init 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.926+0000 7f1fb35c8640 10 monclient: init 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.927+0000 7f1fb35c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:28.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.927+0000 7f1fb35c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:28.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.927+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1fac0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.927+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1fac0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1fac14a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac1239d0 con 0x7f1fac0b0770 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac125cb0 con 0x7f1fac0ac5a0 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac1459a0 con 0x7f1fac14a700 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.928+0000 7f1fb35c8640 10 monclient(hunting): authenticate will time out at 1160.097688s 2024-03-29T10:40:28.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.930+0000 7f1fb1b65640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f1fac14a700 0x7f1fac1db840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60534/0 (socket says 172.21.15.151:60534) 2024-03-29T10:40:28.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.930+0000 7f1fb1b65640 1 -- 172.21.15.151:0/3831982245 learned_addr learned my addr 172.21.15.151:0/3831982245 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:28.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.930+0000 7f1fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f1fac14a700 2024-03-29T10:40:28.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.930+0000 7f1fa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167541939 0 0) 0x7f1fac1459a0 con 0x7f1fac14a700 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:40:28.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f90004eb0 con 0x7f1fac14a700 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4278725120 0 0) 0x7f1f90004eb0 con 0x7f1fac14a700 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fac1459a0 con 0x7f1fac14a700 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1fa80017f0 con 0x7f1fac14a700 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.922 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.931+0000 7f1fa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3636425687 0 0) 0x7f1fac1459a0 con 0x7f1fac14a700 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 >> v1:172.21.15.169:6789/0 conn(0x7f1fac0ac5a0 legacy=0x7f1fac126820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 >> v1:172.21.15.151:6789/0 conn(0x7f1fac0b0770 legacy=0x7f1fac144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 monclient: found mon.c 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac145b30 con 0x7f1fac14a700 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:28.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:28.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:28.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fa8001950 con 0x7f1fac14a700 2024-03-29T10:40:28.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.932+0000 7f1fa2ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f1fa8004aa0 con 0x7f1fac14a700 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fa1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fa1ffb640 10 monclient: dump: 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:28.925 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 10 monclient: _renew_subs 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac126f30 con 0x7f1fac14a700 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 10 monclient: _renew_subs 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac145580 con 0x7f1fac14a700 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.933+0000 7f1fb35c8640 1 librados: init done 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.935+0000 7f1fb35c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.936+0000 7f1fb35c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.936+0000 7f1fb35c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:28.926 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.936+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac0b0b50 con 0x7f1fac14a700 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.937+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f1fa801aa70 con 0x7f1fac14a700 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fb1364640 10 monclient: get_auth_request con 0x7f1f9006dc40 auth_method 0 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f1fa8084b00 con 0x7f1fac14a700 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 10 client.4352.objecter ms_dispatch 0x7f1fac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T10:40:28.927 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.938+0000 7f1fa1ffb640 10 client.4352.objecter ms_handle_connect 0x7f1f9006dc40 2024-03-29T10:40:28.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.947+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fa80503e0 con 0x7f1fac14a700 2024-03-29T10:40:28.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.947+0000 7f1fa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:28.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:28.947+0000 7f1fa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:29.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.276+0000 7f1fb35c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:29.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.276+0000 7f1fb35c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:29.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.276+0000 7f1fb35c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:29.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.276+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fac1281a0 con 0x7f1fac14a700 2024-03-29T10:40:29.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.277+0000 7f1fa1ffb640 1 -- 172.21.15.151:0/3831982245 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f1fa8052310 con 0x7f1fac14a700 2024-03-29T10:40:29.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.277+0000 7f1fa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:29.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.277+0000 7f1fa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:29.261 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:29.261 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:29.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.281+0000 7f1fb35c8640 10 librados: watch_flush enter 2024-03-29T10:40:29.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.281+0000 7f1fb35c8640 10 librados: watch_flush exit 2024-03-29T10:40:29.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T10:40:29.265 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 10 client.4352.objecter successfully canceled tick 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 >> v1:172.21.15.151:6816/1549873000 conn(0x7f1f9006dc40 legacy=0x7f1f90070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 10 monclient: shutdown 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 >> v1:172.21.15.151:6790/0 conn(0x7f1fac14a700 legacy=0x7f1fac1db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 shutdown_connections 2024-03-29T10:40:29.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.282+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 >> 172.21.15.151:0/3831982245 conn(0x7f1fac06b660 msgr2=0x7f1fac06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:29.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.285+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 shutdown_connections 2024-03-29T10:40:29.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.285+0000 7f1fb35c8640 1 -- 172.21.15.151:0/3831982245 wait complete. 2024-03-29T10:40:29.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:29.285+0000 7f1fb35c8640 1 librados: shutdown 2024-03-29T10:40:29.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:36.093 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.254+0000 7f5e49eba640 10 monclient: get_monmap_and_config 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.254+0000 7f5e49eba640 10 monclient: build_initial_monmap 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.254+0000 7f5e49eba640 10 monclient: monmap: 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:36.255200+0000 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:36.255200+0000 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:36.238 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:36.239 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:36.239 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.255+0000 7f5e49eba640 1 Processor -- start 2024-03-29T10:40:36.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.255+0000 7f5e49eba640 1 -- start start 2024-03-29T10:40:36.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.255+0000 7f5e49eba640 10 monclient: init 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-a con 0x7f5e441457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4413f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-b con 0x7f5e440a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:36.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e440aaef0 con 0x7f5e441457f0 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e44145bd0 con 0x7f5e4413f650 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e44145eb0 con 0x7f5e440a9d70 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.256+0000 7f5e49eba640 10 monclient(hunting): authenticate will time out at 1167.426200s 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.257+0000 7f5e43fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f5e440a9d70 0x7f5e4414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:49072/0 (socket says 172.21.15.151:49072) 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.257+0000 7f5e43fff640 1 -- 172.21.15.151:0/842953692 learned_addr learned my addr 172.21.15.151:0/842953692 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208090566 0 0) 0x7f5e44145eb0 con 0x7f5e440a9d70 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003520 con 0x7f5e440a9d70 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242368718 0 0) 0x7f5e2c003520 con 0x7f5e440a9d70 2024-03-29T10:40:36.243 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.258+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 >> v1:172.21.15.169:6789/0 conn(0x7f5e4413f650 legacy=0x7f5e440b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.259+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 >> v1:172.21.15.151:6789/0 conn(0x7f5e441457f0 legacy=0x7f5e4413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44146340 con 0x7f5e440a9d70 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:36.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5e38001670 con 0x7f5e440a9d70 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.260+0000 7f5e427fc640 10 monclient: dump: 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:36.246 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.247 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.262+0000 7f5e49eba640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:40:36.247 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.262+0000 7f5e49eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:36.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.264+0000 7f5e427fc640 1 -- 172.21.15.151:0/842953692 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e380018d0 con 0x7f5e440a9d70 2024-03-29T10:40:36.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.264+0000 7f5e427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:36.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.264+0000 7f5e49eba640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:36.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.264+0000 7f5e49eba640 4 set_mon_vals no callback set 2024-03-29T10:40:36.248 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.265+0000 7f5e49eba640 10 monclient: shutdown 2024-03-29T10:40:36.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.265+0000 7f5e49eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:36.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.265+0000 7f5e49eba640 1 -- 172.21.15.151:0/842953692 >> v1:172.21.15.151:6790/0 conn(0x7f5e440a9d70 legacy=0x7f5e4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:36.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.266+0000 7f5e49eba640 1 -- 172.21.15.151:0/842953692 shutdown_connections 2024-03-29T10:40:36.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.266+0000 7f5e49eba640 1 -- 172.21.15.151:0/842953692 >> 172.21.15.151:0/842953692 conn(0x7f5e44066a00 msgr2=0x7f5e44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:36.249 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.266+0000 7f5e49eba640 1 -- 172.21.15.151:0/842953692 shutdown_connections 2024-03-29T10:40:36.250 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.267+0000 7f5e49eba640 1 -- 172.21.15.151:0/842953692 wait complete. 2024-03-29T10:40:36.253 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.268+0000 7f5e49eba640 10 monclient: build_initial_monmap 2024-03-29T10:40:36.254 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.268+0000 7f5e49eba640 10 monclient: monmap: 2024-03-29T10:40:36.254 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.268+0000 7f5e49eba640 1 librados: starting msgr at 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.268+0000 7f5e49eba640 1 librados: starting objecter 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.268+0000 7f5e49eba640 1 Processor -- start 2024-03-29T10:40:36.255 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 -- start start 2024-03-29T10:40:36.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 librados: setting wanted keys 2024-03-29T10:40:36.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 librados: calling monclient init 2024-03-29T10:40:36.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient: init 2024-03-29T10:40:36.257 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-a con 0x7f5e440a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-c con 0x7f5e4413f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient(hunting): picked mon.noname-b con 0x7f5e441457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4406e9a0 con 0x7f5e440a9d70 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4406fa00 con 0x7f5e441457f0 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4406fce0 con 0x7f5e4413f650 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.269+0000 7f5e49eba640 10 monclient(hunting): authenticate will time out at 1167.439421s 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.271+0000 7f5e42ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f5e4413f650 0x7f5e4406df70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:49650/0 (socket says 172.21.15.151:49650) 2024-03-29T10:40:36.258 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.271+0000 7f5e42ffd640 1 -- 172.21.15.151:0/3233081451 learned_addr learned my addr 172.21.15.151:0/3233081451 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:36.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.274+0000 7f5e2bfff640 10 client.?.objecter ms_handle_connect 0x7f5e4413f650 2024-03-29T10:40:36.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:36.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 client.?.objecter ms_handle_connect 0x7f5e441457f0 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 client.?.objecter ms_handle_connect 0x7f5e440a9d70 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008117003 0 0) 0x7f5e4406fa00 con 0x7f5e441457f0 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:36.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.275+0000 7f5e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e30004eb0 con 0x7f5e441457f0 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467239994 0 0) 0x7f5e4406e9a0 con 0x7f5e440a9d70 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4406fa00 con 0x7f5e440a9d70 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.276+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716621456 0 0) 0x7f5e4406fce0 con 0x7f5e4413f650 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:36.261 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:40:36.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4406e9a0 con 0x7f5e4413f650 2024-03-29T10:40:36.262 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.277+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 876884904 0 0) 0x7f5e30004eb0 con 0x7f5e441457f0 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e4406fce0 con 0x7f5e441457f0 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5e38004b60 con 0x7f5e441457f0 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:36.267 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1053069394 0 0) 0x7f5e4406fa00 con 0x7f5e440a9d70 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e30004eb0 con 0x7f5e440a9d70 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5e3c001620 con 0x7f5e440a9d70 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:36.268 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3937296149 0 0) 0x7f5e4406fce0 con 0x7f5e441457f0 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.279+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 >> v1:172.21.15.151:6790/0 conn(0x7f5e4413f650 legacy=0x7f5e4406df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 >> v1:172.21.15.151:6789/0 conn(0x7f5e440a9d70 legacy=0x7f5e440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 1 monclient: found mon.b 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:36.269 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4406ffc0 con 0x7f5e441457f0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 10 monclient: _finish_auth 0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 5 monclient: authenticate success, global_id 4366 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 10 monclient: _renew_subs 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4406ed10 con 0x7f5e441457f0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 10 monclient: _renew_subs 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e44139860 con 0x7f5e441457f0 2024-03-29T10:40:36.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.280+0000 7f5e49eba640 1 librados: init done 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.281+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e38001250 con 0x7f5e441457f0 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.281+0000 7f5e2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.281+0000 7f5e41ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.281+0000 7f5e49eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.282+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5e38001d20 con 0x7f5e441457f0 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.282+0000 7f5e2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.282+0000 7f5e2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.282+0000 7f5e2bfff640 10 monclient: dump: 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:36.271 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e49eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e49eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f5e3801ade0 con 0x7f5e441457f0 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24005080 con 0x7f5e441457f0 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f5e38084c00 con 0x7f5e441457f0 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e2bfff640 10 client.4366.objecter ms_dispatch 0x7f5e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:36.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e2bfff640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:36.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.283+0000 7f5e2bfff640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:36.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.284+0000 7f5e2bfff640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T10:40:36.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.284+0000 7f5e437fe640 10 monclient: get_auth_request con 0x7f5e3006ddf0 auth_method 0 2024-03-29T10:40:36.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.284+0000 7f5e2bfff640 10 client.4366.objecter ms_handle_connect 0x7f5e3006ddf0 2024-03-29T10:40:36.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.293+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e380504e0 con 0x7f5e441457f0 2024-03-29T10:40:36.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.294+0000 7f5e2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:36.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.294+0000 7f5e2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:36.599 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.614+0000 7f5e49eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:36.599 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.614+0000 7f5e49eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:36.599 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.614+0000 7f5e49eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:36.599 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.614+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e24005330 con 0x7f5e441457f0 2024-03-29T10:40:36.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.616+0000 7f5e2bfff640 1 -- 172.21.15.151:0/3233081451 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f5e38052410 con 0x7f5e441457f0 2024-03-29T10:40:36.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.617+0000 7f5e2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:36.600 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.617+0000 7f5e2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:36.601 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:36.601 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:36.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.622+0000 7f5e49eba640 10 librados: watch_flush enter 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 10 librados: watch_flush exit 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 10 client.4366.objecter successfully canceled tick 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 >> v1:172.21.15.151:6816/1549873000 conn(0x7f5e3006ddf0 legacy=0x7f5e300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 10 monclient: shutdown 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:36.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.624+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 >> v1:172.21.15.169:6789/0 conn(0x7f5e441457f0 legacy=0x7f5e441ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:36.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.625+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 shutdown_connections 2024-03-29T10:40:36.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.625+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 >> 172.21.15.151:0/3233081451 conn(0x7f5e44066a00 msgr2=0x7f5e44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:36.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.627+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 shutdown_connections 2024-03-29T10:40:36.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.627+0000 7f5e49eba640 1 -- 172.21.15.151:0/3233081451 wait complete. 2024-03-29T10:40:36.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:36.627+0000 7f5e49eba640 1 librados: shutdown 2024-03-29T10:40:36.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:43.437 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:43.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.599+0000 7f9329ef2640 10 monclient: get_monmap_and_config 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.599+0000 7f9329ef2640 10 monclient: build_initial_monmap 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.599+0000 7f9329ef2640 10 monclient: monmap: 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:43.600086+0000 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:43.600086+0000 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.600+0000 7f9329ef2640 1 Processor -- start 2024-03-29T10:40:43.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.600+0000 7f9329ef2640 1 -- start start 2024-03-29T10:40:43.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.600+0000 7f9329ef2640 10 monclient: init 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-a con 0x7f93240ada60 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-b con 0x7f932413ed40 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-c con 0x7f93240af230 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9324135bd0 con 0x7f93240ada60 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93240ab440 con 0x7f93240af230 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932413fc40 con 0x7f932413ed40 2024-03-29T10:40:43.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:43.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.601+0000 7f9329ef2640 10 monclient(hunting): authenticate will time out at 1174.770816s 2024-03-29T10:40:43.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f9323fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f932413ed40 0x7f932413f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:43272/0 (socket says 172.21.15.151:43272) 2024-03-29T10:40:43.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f9323fff640 1 -- 172.21.15.151:0/3324783589 learned_addr learned my addr 172.21.15.151:0/3324783589 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807658728 0 0) 0x7f9324135bd0 con 0x7f93240ada60 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.604+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314003520 con 0x7f93240ada60 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343458228 0 0) 0x7f932413fc40 con 0x7f932413ed40 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324135bd0 con 0x7f932413ed40 2024-03-29T10:40:43.589 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098582615 0 0) 0x7f9314003520 con 0x7f93240ada60 2024-03-29T10:40:43.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 >> v1:172.21.15.151:6790/0 conn(0x7f932413ed40 legacy=0x7f932413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.605+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 >> v1:172.21.15.169:6789/0 conn(0x7f93240af230 legacy=0x7f932414b860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:43.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:43.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 1 monclient: found mon.noname-a 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93240ace10 con 0x7f93240ada60 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.606+0000 7f93237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f9329ef2640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f9329ef2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9318001610 con 0x7f93240ada60 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f93237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f93237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f93237fe640 10 monclient: dump: 2024-03-29T10:40:43.592 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.607+0000 7f9329ef2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:43.595 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.608+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9318001770 con 0x7f93240ada60 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f93237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f9329ef2640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f9329ef2640 4 set_mon_vals no callback set 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f93237fe640 1 -- 172.21.15.151:0/3324783589 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9318001bc0 con 0x7f93240ada60 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f93237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f93237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f93237fe640 10 monclient: dump: 2024-03-29T10:40:43.596 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f9329ef2640 10 monclient: shutdown 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f9329ef2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.609+0000 7f9329ef2640 1 -- 172.21.15.151:0/3324783589 >> v1:172.21.15.151:6789/0 conn(0x7f93240ada60 legacy=0x7f93240ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.597 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.610+0000 7f9329ef2640 1 -- 172.21.15.151:0/3324783589 shutdown_connections 2024-03-29T10:40:43.598 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.610+0000 7f9329ef2640 1 -- 172.21.15.151:0/3324783589 >> 172.21.15.151:0/3324783589 conn(0x7f93240673e0 msgr2=0x7f93240b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:43.598 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.610+0000 7f9329ef2640 1 -- 172.21.15.151:0/3324783589 shutdown_connections 2024-03-29T10:40:43.598 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.610+0000 7f9329ef2640 1 -- 172.21.15.151:0/3324783589 wait complete. 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.611+0000 7f9329ef2640 10 monclient: build_initial_monmap 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.611+0000 7f9329ef2640 10 monclient: monmap: 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.604 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.611+0000 7f9329ef2640 1 librados: starting msgr at 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.611+0000 7f9329ef2640 1 librados: starting objecter 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.612+0000 7f9329ef2640 1 Processor -- start 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.612+0000 7f9329ef2640 1 -- start start 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.612+0000 7f9329ef2640 1 librados: setting wanted keys 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.612+0000 7f9329ef2640 1 librados: calling monclient init 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.612+0000 7f9329ef2640 10 monclient: init 2024-03-29T10:40:43.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-c con 0x7f93240ada60 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-b con 0x7f93240af230 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient(hunting): picked mon.noname-a con 0x7f932413ed40 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9324135d60 con 0x7f932413ed40 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9324126390 con 0x7f93240af230 2024-03-29T10:40:43.606 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932413bb80 con 0x7f93240ada60 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.613+0000 7f9329ef2640 10 monclient(hunting): authenticate will time out at 1174.782716s 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f9328ef0640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f93240ada60 0x7f9324139d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:53934/0 (socket says 172.21.15.151:53934) 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f9328ef0640 1 -- 172.21.15.151:0/2040670720 learned_addr learned my addr 172.21.15.151:0/2040670720 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 client.?.objecter ms_handle_connect 0x7f93240ada60 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927016075 0 0) 0x7f932413bb80 con 0x7f93240ada60 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.615+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9308004eb0 con 0x7f93240ada60 2024-03-29T10:40:43.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226758721 0 0) 0x7f9308004eb0 con 0x7f93240ada60 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f932413bb80 con 0x7f93240ada60 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9318001610 con 0x7f93240ada60 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 10 monclient(hunting): dump: 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:43.608 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322204150 0 0) 0x7f932413bb80 con 0x7f93240ada60 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.616+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 >> v1:172.21.15.169:6789/0 conn(0x7f93240af230 legacy=0x7f932413db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.617+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 >> v1:172.21.15.151:6789/0 conn(0x7f932413ed40 legacy=0x7f93241e2130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 10 monclient: _finish_hunting 0 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 1 monclient: found mon.c 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932413a460 con 0x7f93240ada60 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 10 monclient: _finish_auth 0 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 10 monclient: _check_auth_tickets 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.618+0000 7f93217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:43.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 10 monclient: _renew_subs 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9324139520 con 0x7f93240ada60 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 10 monclient: _renew_subs 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f932413a9d0 con 0x7f93240ada60 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9329ef2640 1 librados: init done 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93180038a0 con 0x7f93240ada60 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9318003cf0 con 0x7f93240ada60 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f93217fa640 10 monclient: dump: 2024-03-29T10:40:43.610 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:43.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.619+0000 7f9322ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.621+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f9318003e50 con 0x7f93240ada60 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.621+0000 7f9329ef2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.621+0000 7f9329ef2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.621+0000 7f9329ef2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.621+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9304005080 con 0x7f93240ada60 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f9318084c80 con 0x7f93240ada60 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f93217fa640 10 client.4364.objecter ms_dispatch 0x7f9324000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f93217fa640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:43.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f93217fa640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:43.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f93217fa640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T10:40:43.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.622+0000 7f9323fff640 10 monclient: get_auth_request con 0x7f930806dbe0 auth_method 0 2024-03-29T10:40:43.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.623+0000 7f93217fa640 10 client.4364.objecter ms_handle_connect 0x7f930806dbe0 2024-03-29T10:40:43.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.630+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9318050560 con 0x7f93240ada60 2024-03-29T10:40:43.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.630+0000 7f93217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.630+0000 7f93217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:43.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.950+0000 7f9329ef2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.950+0000 7f9329ef2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.950+0000 7f9329ef2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:40:43.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.950+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9304005330 con 0x7f93240ada60 2024-03-29T10:40:43.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.952+0000 7f93217fa640 1 -- 172.21.15.151:0/2040670720 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f9318052490 con 0x7f93240ada60 2024-03-29T10:40:43.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.952+0000 7f93217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.952+0000 7f93217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:43.938 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:43.939 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:43.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.956+0000 7f9329ef2640 10 librados: watch_flush enter 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.956+0000 7f9329ef2640 10 librados: watch_flush exit 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.956+0000 7f9329ef2640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.956+0000 7f9329ef2640 10 client.4364.objecter successfully canceled tick 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 >> v1:172.21.15.151:6816/1549873000 conn(0x7f930806dbe0 legacy=0x7f93080700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 10 monclient: shutdown 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 >> v1:172.21.15.151:6790/0 conn(0x7f93240ada60 legacy=0x7f9324139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 shutdown_connections 2024-03-29T10:40:43.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.957+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 >> 172.21.15.151:0/2040670720 conn(0x7f93240673e0 msgr2=0x7f93240b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:43.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.959+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 shutdown_connections 2024-03-29T10:40:43.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.959+0000 7f9329ef2640 1 -- 172.21.15.151:0/2040670720 wait complete. 2024-03-29T10:40:43.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:43.959+0000 7f9329ef2640 1 librados: shutdown 2024-03-29T10:40:43.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:50.767 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.945+0000 7f8597446640 10 monclient: get_monmap_and_config 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.945+0000 7f8597446640 10 monclient: build_initial_monmap 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.945+0000 7f8597446640 10 monclient: monmap: 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:50.946077+0000 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:50.946077+0000 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:50.929 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:50.930 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:50.930 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.946+0000 7f8597446640 1 Processor -- start 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.947+0000 7f8597446640 1 -- start start 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.947+0000 7f8597446640 10 monclient: init 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient(hunting): picked mon.noname-c con 0x7f85900b4920 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:50.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient(hunting): picked mon.noname-a con 0x7f85900b4200 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient(hunting): picked mon.noname-b con 0x7f859013f9a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900b0930 con 0x7f85900b4200 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900af460 con 0x7f85900b4920 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900aeda0 con 0x7f859013f9a0 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:50.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.948+0000 7f8597446640 10 monclient(hunting): authenticate will time out at 1182.117957s 2024-03-29T10:40:50.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.952+0000 7f8595c43640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f85900b4200 0x7f85900b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59825/0 (socket says 172.21.15.151:59825) 2024-03-29T10:40:50.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.952+0000 7f8595c43640 1 -- 172.21.15.151:0/3816651075 learned_addr learned my addr 172.21.15.151:0/3816651075 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265707812 0 0) 0x7f85900b0930 con 0x7f85900b4200 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:40:50.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.954+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f857c003520 con 0x7f85900b4200 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980971407 0 0) 0x7f857c003520 con 0x7f85900b4200 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 >> v1:172.21.15.151:6790/0 conn(0x7f859013f9a0 legacy=0x7f859014d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 >> v1:172.21.15.169:6789/0 conn(0x7f85900b4920 legacy=0x7f859013f290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: _finish_hunting 0 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 monclient: found mon.noname-a 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900af980 con 0x7f85900b4200 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: _finish_auth 0 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: _check_auth_tickets 2024-03-29T10:40:50.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8580001690 con 0x7f85900b4200 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8597446640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8597446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.955+0000 7f8595442640 10 monclient: dump: 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:50.940 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:50.941 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8580001860 con 0x7f85900b4200 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 1 -- 172.21.15.151:0/3816651075 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8580001cb0 con 0x7f85900b4200 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8595442640 10 monclient: dump: 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.942 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8597446640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.956+0000 7f8597446640 4 set_mon_vals no callback set 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.957+0000 7f8597446640 10 monclient: shutdown 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.957+0000 7f8597446640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:50.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.957+0000 7f8597446640 1 -- 172.21.15.151:0/3816651075 >> v1:172.21.15.151:6789/0 conn(0x7f85900b4200 legacy=0x7f85900b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.960+0000 7f8597446640 1 -- 172.21.15.151:0/3816651075 shutdown_connections 2024-03-29T10:40:50.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.960+0000 7f8597446640 1 -- 172.21.15.151:0/3816651075 >> 172.21.15.151:0/3816651075 conn(0x7f859006d230 msgr2=0x7f8590066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:50.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.960+0000 7f8597446640 1 -- 172.21.15.151:0/3816651075 shutdown_connections 2024-03-29T10:40:50.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.961+0000 7f8597446640 1 -- 172.21.15.151:0/3816651075 wait complete. 2024-03-29T10:40:50.945 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 10 monclient: build_initial_monmap 2024-03-29T10:40:50.945 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 10 monclient: monmap: 2024-03-29T10:40:50.945 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:50.945 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:50.946 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 1 librados: starting msgr at 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 1 librados: starting objecter 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 1 Processor -- start 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.962+0000 7f8597446640 1 -- start start 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 1 librados: setting wanted keys 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 1 librados: calling monclient init 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient: init 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:50.948 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient(hunting): picked mon.noname-b con 0x7f85900b4200 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient(hunting): picked mon.noname-a con 0x7f85900b4920 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient(hunting): picked mon.noname-c con 0x7f859013f9a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8590071eb0 con 0x7f85900b4920 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85901e02c0 con 0x7f85900b4200 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8590070430 con 0x7f859013f9a0 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:50.949 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.963+0000 7f8597446640 10 monclient(hunting): authenticate will time out at 1182.133299s 2024-03-29T10:40:50.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8596444640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f85900b4200 0x7f85901df4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:59995/0 (socket says 172.21.15.151:59995) 2024-03-29T10:40:50.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8596444640 1 -- 172.21.15.151:0/448767480 learned_addr learned my addr 172.21.15.151:0/448767480 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:50.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 client.?.objecter ms_handle_connect 0x7f85900b4200 2024-03-29T10:40:50.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075221337 0 0) 0x7f85901e02c0 con 0x7f85900b4200 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.967+0000 7f8586ffd640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8574004eb0 con 0x7f85900b4200 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194373244 0 0) 0x7f8574004eb0 con 0x7f85900b4200 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85901e02c0 con 0x7f85900b4200 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f858c001870 con 0x7f85900b4200 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:50.953 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.968+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082858161 0 0) 0x7f85901e02c0 con 0x7f85900b4200 2024-03-29T10:40:50.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.969+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 >> v1:172.21.15.151:6790/0 conn(0x7f859013f9a0 legacy=0x7f85901e66f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 >> v1:172.21.15.151:6789/0 conn(0x7f85900b4920 legacy=0x7f85901dfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 1 monclient: found mon.b 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590070710 con 0x7f85900b4200 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 10 monclient: _renew_subs 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:50.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85900acfa0 con 0x7f85900b4200 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 10 monclient: _renew_subs 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f859006f1f0 con 0x7f85900b4200 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.970+0000 7f8597446640 1 librados: init done 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.972+0000 7f8597446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.972+0000 7f8597446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.972+0000 7f8597446640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:50.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.972+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859004f730 con 0x7f85900b4200 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f858c0019d0 con 0x7f85900b4200 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f858c004ae0 con 0x7f85900b4200 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8586ffd640 10 monclient: dump: 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:50.959 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:50.960 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:50.960 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:50.960 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:50.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.974+0000 7f8594c41640 4 set_mon_vals no callback set 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.978+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f858c01aab0 con 0x7f85900b4200 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f858c084aa0 con 0x7f85900b4200 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8595c43640 10 monclient: get_auth_request con 0x7f857406dc40 auth_method 0 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 10 client.4369.objecter ms_dispatch 0x7f8590000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:50.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:50.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T10:40:50.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.979+0000 7f8586ffd640 10 client.4369.objecter ms_handle_connect 0x7f857406dc40 2024-03-29T10:40:50.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.983+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f858c0501b0 con 0x7f85900b4200 2024-03-29T10:40:50.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.983+0000 7f8586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:50.983+0000 7f8586ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:51.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.304+0000 7f8597446640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.304+0000 7f8597446640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.304+0000 7f8597446640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:40:51.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.304+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85900b0820 con 0x7f85900b4200 2024-03-29T10:40:51.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.308+0000 7f8586ffd640 1 -- 172.21.15.151:0/448767480 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f858c0522b0 con 0x7f85900b4200 2024-03-29T10:40:51.292 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:51.292 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:51.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.308+0000 7f8586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.308+0000 7f8586ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:51.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.312+0000 7f8597446640 10 librados: watch_flush enter 2024-03-29T10:40:51.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.313+0000 7f8597446640 10 librados: watch_flush exit 2024-03-29T10:40:51.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.313+0000 7f8597446640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T10:40:51.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.313+0000 7f8597446640 10 client.4369.objecter successfully canceled tick 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 >> v1:172.21.15.151:6816/1549873000 conn(0x7f857406dc40 legacy=0x7f8574070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 10 monclient: shutdown 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 >> v1:172.21.15.169:6789/0 conn(0x7f85900b4200 legacy=0x7f85901df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 shutdown_connections 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.314+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 >> 172.21.15.151:0/448767480 conn(0x7f859006d230 msgr2=0x7f85900b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:51.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.315+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 shutdown_connections 2024-03-29T10:40:51.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.316+0000 7f8597446640 1 -- 172.21.15.151:0/448767480 wait complete. 2024-03-29T10:40:51.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:51.319+0000 7f8597446640 1 librados: shutdown 2024-03-29T10:40:51.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:58.127 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.291+0000 7f7987fff640 10 monclient: get_monmap_and_config 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.291+0000 7f7987fff640 10 monclient: build_initial_monmap 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.291+0000 7f7987fff640 10 monclient: monmap: 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:40:58.293691+0000 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:40:58.293691+0000 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:58.277 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.278 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:58.278 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:40:58.278 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:40:58.278 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.292+0000 7f7987fff640 1 Processor -- start 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.292+0000 7f7987fff640 1 -- start start 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.292+0000 7f7987fff640 10 monclient: init 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-a con 0x7f79880a9750 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:58.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-c con 0x7f79880ae220 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-b con 0x7f798813b9d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7988142490 con 0x7f79880a9750 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880ac130 con 0x7f79880ae220 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880ac410 con 0x7f798813b9d0 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:58.281 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.293+0000 7f7987fff640 10 monclient(hunting): authenticate will time out at 1189.464362s 2024-03-29T10:40:58.282 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.298+0000 7f79877fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f798813b9d0 0x7f7988149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:59673/0 (socket says 172.21.15.151:59673) 2024-03-29T10:40:58.282 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.298+0000 7f79877fe640 1 -- 172.21.15.151:0/1949839887 learned_addr learned my addr 172.21.15.151:0/1949839887 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487114494 0 0) 0x7f79880ac410 con 0x7f798813b9d0 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.301+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7970003520 con 0x7f798813b9d0 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693366974 0 0) 0x7f7988142490 con 0x7f79880a9750 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79880ac410 con 0x7f79880a9750 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3806665070 0 0) 0x7f79880ac410 con 0x7f79880a9750 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 >> v1:172.21.15.151:6790/0 conn(0x7f798813b9d0 legacy=0x7f7988149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 >> v1:172.21.15.169:6789/0 conn(0x7f79880ae220 legacy=0x7f798813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 monclient: found mon.noname-a 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79880afa40 con 0x7f79880a9750 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7978001670 con 0x7f79880a9750 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: dump: 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7987fff640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7987fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79780017d0 con 0x7f79880a9750 2024-03-29T10:40:58.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 1 -- 172.21.15.151:0/1949839887 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7978001b60 con 0x7f79880a9750 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7985ffb640 10 monclient: dump: 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7987fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.302+0000 7f7987fff640 4 set_mon_vals no callback set 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.303+0000 7f7987fff640 10 monclient: shutdown 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.303+0000 7f7987fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.303+0000 7f7987fff640 1 -- 172.21.15.151:0/1949839887 >> v1:172.21.15.151:6789/0 conn(0x7f79880a9750 legacy=0x7f7988141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.303+0000 7f7987fff640 1 -- 172.21.15.151:0/1949839887 shutdown_connections 2024-03-29T10:40:58.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.303+0000 7f7987fff640 1 -- 172.21.15.151:0/1949839887 >> 172.21.15.151:0/1949839887 conn(0x7f79880673e0 msgr2=0x7f79880b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.304+0000 7f7987fff640 1 -- 172.21.15.151:0/1949839887 shutdown_connections 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.304+0000 7f7987fff640 1 -- 172.21.15.151:0/1949839887 wait complete. 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.305+0000 7f7987fff640 10 monclient: build_initial_monmap 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.305+0000 7f7987fff640 10 monclient: monmap: 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:40:58.293 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.305+0000 7f7987fff640 1 librados: starting msgr at 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.305+0000 7f7987fff640 1 librados: starting objecter 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.305+0000 7f7987fff640 1 Processor -- start 2024-03-29T10:40:58.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.307+0000 7f7987fff640 1 -- start start 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.307+0000 7f7987fff640 1 librados: setting wanted keys 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.307+0000 7f7987fff640 1 librados: calling monclient init 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.307+0000 7f7987fff640 10 monclient: init 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-c con 0x7f79880a9750 addr v1:172.21.15.151:6790/0 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-b con 0x7f79880ae220 addr v1:172.21.15.169:6789/0 2024-03-29T10:40:58.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient(hunting): picked mon.noname-a con 0x7f798813b9d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798813eec0 con 0x7f798813b9d0 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798806d8c0 con 0x7f79880ae220 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880b2f80 con 0x7f79880a9750 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.308+0000 7f7987fff640 10 monclient(hunting): authenticate will time out at 1189.479046s 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.312+0000 7f79867fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f79880ae220 0x7f79881e2f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:55919/0 (socket says 172.21.15.151:55919) 2024-03-29T10:40:58.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.312+0000 7f79867fc640 1 -- 172.21.15.151:0/425504719 learned_addr learned my addr 172.21.15.151:0/425504719 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 client.?.objecter ms_handle_connect 0x7f798813b9d0 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106363395 0 0) 0x7f798813eec0 con 0x7f798813b9d0 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7974004eb0 con 0x7f798813b9d0 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 client.?.objecter ms_handle_connect 0x7f79880ae220 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.313+0000 7f796ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1178963066 0 0) 0x7f7974004eb0 con 0x7f798813b9d0 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f798813eec0 con 0x7f798813b9d0 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f797c001890 con 0x7f798813b9d0 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): dump: 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.299 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492457153 0 0) 0x7f798806d8c0 con 0x7f79880ae220 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:40:58.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7974004eb0 con 0x7f79880ae220 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3954205930 0 0) 0x7f798813eec0 con 0x7f798813b9d0 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.314+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 >> v1:172.21.15.151:6790/0 conn(0x7f79880a9750 legacy=0x7f798813d700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 >> v1:172.21.15.169:6789/0 conn(0x7f79880ae220 legacy=0x7f79881e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 1 monclient: found mon.a 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798806dce0 con 0x7f798813b9d0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 10 monclient: _finish_auth 0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f796ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 10 monclient: _renew_subs 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798806df30 con 0x7f798813b9d0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 10 monclient: _renew_subs 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79880b2bd0 con 0x7f798813b9d0 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.315+0000 7f7987fff640 1 librados: init done 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.317+0000 7f796dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.317+0000 7f796dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.317+0000 7f796dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.303 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.317+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798804f730 con 0x7f798813b9d0 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.320+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f797c0047e0 con 0x7f798813b9d0 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.320+0000 7f796ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.320+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f797c004bf0 con 0x7f798813b9d0 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.320+0000 7f79857fa640 4 set_mon_vals no callback set 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.321+0000 7f796ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.321+0000 7f796ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:58.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.321+0000 7f796ffff640 10 monclient: dump: 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:40:58.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.322+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f797c004e10 con 0x7f798813b9d0 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.322+0000 7f7986ffd640 10 monclient: get_auth_request con 0x7f797406dde0 auth_method 0 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f797c084ac0 con 0x7f798813b9d0 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 10 client.4362.objecter ms_dispatch 0x7f7988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T10:40:58.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.323+0000 7f796ffff640 10 client.4362.objecter ms_handle_connect 0x7f797406dde0 2024-03-29T10:40:58.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.327+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f797c050420 con 0x7f798813b9d0 2024-03-29T10:40:58.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.327+0000 7f796ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.327+0000 7f796ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:58.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.650+0000 7f796dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.650+0000 7f796dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.650+0000 7f796dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:40:58.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.650+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79880abb70 con 0x7f798813b9d0 2024-03-29T10:40:58.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.652+0000 7f796ffff640 1 -- 172.21.15.151:0/425504719 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f797c052350 con 0x7f798813b9d0 2024-03-29T10:40:58.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.652+0000 7f796ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.652+0000 7f796ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:58.639 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:40:58.639 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:58.642 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.658+0000 7f796dffb640 10 librados: watch_flush enter 2024-03-29T10:40:58.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 10 librados: watch_flush exit 2024-03-29T10:40:58.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T10:40:58.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 10 client.4362.objecter successfully canceled tick 2024-03-29T10:40:58.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 >> v1:172.21.15.151:6816/1549873000 conn(0x7f797406dde0 legacy=0x7f79740702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 10 monclient: shutdown 2024-03-29T10:40:58.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:58.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.659+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 >> v1:172.21.15.151:6789/0 conn(0x7f798813b9d0 legacy=0x7f79881e66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.661+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 shutdown_connections 2024-03-29T10:40:58.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.661+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 >> 172.21.15.151:0/425504719 conn(0x7f79880673e0 msgr2=0x7f7988069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:58.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.662+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 shutdown_connections 2024-03-29T10:40:58.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.662+0000 7f796dffb640 1 -- 172.21.15.151:0/425504719 wait complete. 2024-03-29T10:40:58.647 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:40:58.663+0000 7f796dffb640 1 librados: shutdown 2024-03-29T10:40:58.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:05.474 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:05.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.643+0000 7f89920e4640 10 monclient: get_monmap_and_config 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.644+0000 7f89920e4640 10 monclient: build_initial_monmap 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.644+0000 7f89920e4640 10 monclient: monmap: 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:05.644962+0000 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:05.644962+0000 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:05.628 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:05.629 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.645+0000 7f89920e4640 1 Processor -- start 2024-03-29T10:41:05.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.645+0000 7f89920e4640 1 -- start start 2024-03-29T10:41:05.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.645+0000 7f89920e4640 10 monclient: init 2024-03-29T10:41:05.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.646+0000 7f89920e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:05.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.646+0000 7f89920e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:05.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.646+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-a con 0x7f898c0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-c con 0x7f898c14a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-b con 0x7f898c0ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c0aba70 con 0x7f898c0b0770 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c0aecb0 con 0x7f898c14a700 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c0af150 con 0x7f898c0ac5a0 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:05.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.651+0000 7f89920e4640 10 monclient(hunting): authenticate will time out at 1196.821324s 2024-03-29T10:41:05.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898affd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f898c14a700 0x7f898c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:47305/0 (socket says 172.21.15.151:47305) 2024-03-29T10:41:05.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898affd640 1 -- 172.21.15.151:0/2878606490 learned_addr learned my addr 172.21.15.151:0/2878606490 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:05.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499714357 0 0) 0x7f898c0aecb0 con 0x7f898c14a700 2024-03-29T10:41:05.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:41:05.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.652+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8974003520 con 0x7f898c14a700 2024-03-29T10:41:05.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.655+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164588357 0 0) 0x7f8974003520 con 0x7f898c14a700 2024-03-29T10:41:05.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.655+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 >> v1:172.21.15.151:6790/0 conn(0x7f898c0ac5a0 legacy=0x7f898c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:05.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.655+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 >> v1:172.21.15.151:6789/0 conn(0x7f898c0b0770 legacy=0x7f898c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.640 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.655+0000 7f898a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.655+0000 7f898a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0ac9d0 con 0x7f898c14a700 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f89800016b0 con 0x7f898c14a700 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.641 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: dump: 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.642 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8980001880 con 0x7f898c14a700 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 1 -- 172.21.15.151:0/2878606490 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8980001cd0 con 0x7f898c14a700 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f898a7fc640 10 monclient: dump: 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.643 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f89920e4640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f89920e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:05.644 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.656+0000 7f89920e4640 4 set_mon_vals no callback set 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.657+0000 7f89920e4640 10 monclient: shutdown 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.657+0000 7f89920e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.657+0000 7f89920e4640 1 -- 172.21.15.151:0/2878606490 >> v1:172.21.15.169:6789/0 conn(0x7f898c14a700 legacy=0x7f898c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.658+0000 7f89920e4640 1 -- 172.21.15.151:0/2878606490 shutdown_connections 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.658+0000 7f89920e4640 1 -- 172.21.15.151:0/2878606490 >> 172.21.15.151:0/2878606490 conn(0x7f898c06b660 msgr2=0x7f898c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:05.645 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.658+0000 7f89920e4640 1 -- 172.21.15.151:0/2878606490 shutdown_connections 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.662+0000 7f89920e4640 1 -- 172.21.15.151:0/2878606490 wait complete. 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.663+0000 7f89920e4640 10 monclient: build_initial_monmap 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.663+0000 7f89920e4640 10 monclient: monmap: 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:05.646 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.647 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.663+0000 7f89920e4640 1 librados: starting msgr at 2024-03-29T10:41:05.648 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.663+0000 7f89920e4640 1 librados: starting objecter 2024-03-29T10:41:05.648 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.663+0000 7f89920e4640 1 Processor -- start 2024-03-29T10:41:05.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.666+0000 7f89920e4640 1 -- start start 2024-03-29T10:41:05.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.666+0000 7f89920e4640 1 librados: setting wanted keys 2024-03-29T10:41:05.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.666+0000 7f89920e4640 1 librados: calling monclient init 2024-03-29T10:41:05.650 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.666+0000 7f89920e4640 10 monclient: init 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.667+0000 7f89920e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.667+0000 7f89920e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.667+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-b con 0x7f898c0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-a con 0x7f898c0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 10 monclient(hunting): picked mon.noname-c con 0x7f898c14a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c1decc0 con 0x7f898c0b0770 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c1282b0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c128590 con 0x7f898c14a700 2024-03-29T10:41:05.652 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:05.653 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.668+0000 7f89920e4640 10 monclient(hunting): authenticate will time out at 1196.837914s 2024-03-29T10:41:05.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.671+0000 7f898b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f898c0ac5a0 0x7f898c126f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:47451/0 (socket says 172.21.15.151:47451) 2024-03-29T10:41:05.655 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.672+0000 7f898b7fe640 1 -- 172.21.15.151:0/80541704 learned_addr learned my addr 172.21.15.151:0/80541704 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 client.?.objecter ms_handle_connect 0x7f898c0ac5a0 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 client.?.objecter ms_handle_connect 0x7f898c14a700 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721901040 0 0) 0x7f898c1282b0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.658 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8978004eb0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207034775 0 0) 0x7f898c128590 con 0x7f898c14a700 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.674+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f898c1282b0 con 0x7f898c14a700 2024-03-29T10:41:05.659 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.675+0000 7f8973fff640 10 client.?.objecter ms_handle_connect 0x7f898c0b0770 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.675+0000 7f8973fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006142461 0 0) 0x7f8978004eb0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f898c128590 con 0x7f898c0ac5a0 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f897c001990 con 0x7f898c0ac5a0 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): dump: 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.660 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.661 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210539474 0 0) 0x7f898c1282b0 con 0x7f898c14a700 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8978004eb0 con 0x7f898c14a700 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8984001810 con 0x7f898c14a700 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient(hunting): dump: 2024-03-29T10:41:05.663 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862692848 0 0) 0x7f898c128590 con 0x7f898c0ac5a0 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 >> v1:172.21.15.151:6790/0 conn(0x7f898c14a700 legacy=0x7f898c1e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 >> v1:172.21.15.151:6789/0 conn(0x7f898c0b0770 legacy=0x7f898c1df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.664 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 monclient: found mon.b 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c1239d0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.676+0000 7f8973fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f897c003610 con 0x7f898c0ac5a0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8989ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f897c0039a0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.665 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.677+0000 7f8973fff640 10 monclient: dump: 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 5 monclient: authenticate success, global_id 4378 2024-03-29T10:41:05.666 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 10 monclient: _renew_subs 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c1df860 con 0x7f898c0ac5a0 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 10 monclient: _renew_subs 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c126380 con 0x7f898c0ac5a0 2024-03-29T10:41:05.667 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.678+0000 7f89920e4640 1 librados: init done 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.682+0000 7f8971ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.682+0000 7f8971ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.682+0000 7f8971ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.682+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c123bc0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.682+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f897c003b00 con 0x7f898c0ac5a0 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.683+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f897c0849d0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.668 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.683+0000 7f8973fff640 10 client.4378.objecter ms_dispatch 0x7f898c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:05.669 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.683+0000 7f8973fff640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:05.669 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.683+0000 7f8973fff640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:05.669 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.683+0000 7f8973fff640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T10:41:05.669 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.686+0000 7f898affd640 10 monclient: get_auth_request con 0x7f897806ded0 auth_method 0 2024-03-29T10:41:05.670 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.687+0000 7f8973fff640 10 client.4378.objecter ms_handle_connect 0x7f897806ded0 2024-03-29T10:41:05.674 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.691+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f897c0502b0 con 0x7f898c0ac5a0 2024-03-29T10:41:05.675 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.691+0000 7f8973fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.675 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:05.691+0000 7f8973fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:06.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.043+0000 7f8971ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:06.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.043+0000 7f8971ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:06.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.043+0000 7f8971ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:06.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.043+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f898c1df9f0 con 0x7f898c0ac5a0 2024-03-29T10:41:06.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.044+0000 7f8973fff640 1 -- 172.21.15.151:0/80541704 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f897c0521e0 con 0x7f898c0ac5a0 2024-03-29T10:41:06.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.044+0000 7f8973fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:06.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.044+0000 7f8973fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:06.032 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:06.032 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:06.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.048+0000 7f8971ffb640 10 librados: watch_flush enter 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.048+0000 7f8971ffb640 10 librados: watch_flush exit 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.048+0000 7f8971ffb640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.048+0000 7f8971ffb640 10 client.4378.objecter successfully canceled tick 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.049+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 >> v1:172.21.15.151:6816/1549873000 conn(0x7f897806ded0 legacy=0x7f8978070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.049+0000 7f8971ffb640 10 monclient: shutdown 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.049+0000 7f8971ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.049+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 >> v1:172.21.15.169:6789/0 conn(0x7f898c0ac5a0 legacy=0x7f898c126f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.050+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 shutdown_connections 2024-03-29T10:41:06.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.050+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 >> 172.21.15.151:0/80541704 conn(0x7f898c06b660 msgr2=0x7f898c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:06.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.052+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 shutdown_connections 2024-03-29T10:41:06.038 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.055+0000 7f8971ffb640 1 -- 172.21.15.151:0/80541704 wait complete. 2024-03-29T10:41:06.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:06.059+0000 7f8971ffb640 1 librados: shutdown 2024-03-29T10:41:06.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:12.871 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.036+0000 7f807e0d9640 10 monclient: get_monmap_and_config 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.036+0000 7f807e0d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.036+0000 7f807e0d9640 10 monclient: monmap: 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:13.037452+0000 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:13.037452+0000 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:13.022 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:13.023 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:13.023 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.038+0000 7f807e0d9640 1 Processor -- start 2024-03-29T10:41:13.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.038+0000 7f807e0d9640 1 -- start start 2024-03-29T10:41:13.023 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.038+0000 7f807e0d9640 10 monclient: init 2024-03-29T10:41:13.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:13.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:13.027 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f80780a9930 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8078149dc0 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f807814ff40 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80780aae60 con 0x7f8078149dc0 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80780b4310 con 0x7f807814ff40 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80780afdd0 con 0x7f80780a9930 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.039+0000 7f807e0d9640 10 monclient(hunting): authenticate will time out at 1204.208815s 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.040+0000 7f8076ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8078149dc0 0x7f807814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59246/0 (socket says 172.21.15.151:59246) 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.040+0000 7f8076ffd640 1 -- 172.21.15.151:0/2608262909 learned_addr learned my addr 172.21.15.151:0/2608262909 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.043+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357932297 0 0) 0x7f80780aae60 con 0x7f8078149dc0 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.043+0000 7f80767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.043+0000 7f80767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:13.028 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:41:13.029 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8060003520 con 0x7f8078149dc0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705392496 0 0) 0x7f8060003520 con 0x7f8078149dc0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 >> v1:172.21.15.151:6790/0 conn(0x7f80780a9930 legacy=0x7f8078143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 >> v1:172.21.15.169:6789/0 conn(0x7f807814ff40 legacy=0x7f8078152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 monclient: found mon.noname-a 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807814a950 con 0x7f8078149dc0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:13.032 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.044+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8070001a10 con 0x7f8078149dc0 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: dump: 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:13.033 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f807e0d9640 5 monclient: authenticate success, global_id 4368 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f807e0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8070001be0 con 0x7f8078149dc0 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 1 -- 172.21.15.151:0/2608262909 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8070002030 con 0x7f8078149dc0 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f80767fc640 10 monclient: dump: 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:13.034 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f807e0d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.045+0000 7f807e0d9640 4 set_mon_vals no callback set 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 10 monclient: shutdown 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 1 -- 172.21.15.151:0/2608262909 >> v1:172.21.15.151:6789/0 conn(0x7f8078149dc0 legacy=0x7f807814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 1 -- 172.21.15.151:0/2608262909 shutdown_connections 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 1 -- 172.21.15.151:0/2608262909 >> 172.21.15.151:0/2608262909 conn(0x7f80780b35e0 msgr2=0x7f80780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:13.035 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 1 -- 172.21.15.151:0/2608262909 shutdown_connections 2024-03-29T10:41:13.044 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.046+0000 7f807e0d9640 1 -- 172.21.15.151:0/2608262909 wait complete. 2024-03-29T10:41:13.044 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.047+0000 7f807e0d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.047+0000 7f807e0d9640 10 monclient: monmap: 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 librados: starting msgr at 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 librados: starting objecter 2024-03-29T10:41:13.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 Processor -- start 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 -- start start 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 librados: setting wanted keys 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 1 librados: calling monclient init 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.048+0000 7f807e0d9640 10 monclient: init 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f80780a9930 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f8078149dc0 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f807814ff40 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807806f5e0 con 0x7f8078149dc0 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80780b2e90 con 0x7f80780a9930 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8078138960 con 0x7f807814ff40 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:13.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.049+0000 7f807e0d9640 10 monclient(hunting): authenticate will time out at 1204.219429s 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.052+0000 7f8076ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8078149dc0 0x7f807813d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59904/0 (socket says 172.21.15.151:59904) 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.052+0000 7f8076ffd640 1 -- 172.21.15.151:0/2556939424 learned_addr learned my addr 172.21.15.151:0/2556939424 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.052+0000 7f805ffff640 10 client.?.objecter ms_handle_connect 0x7f8078149dc0 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.052+0000 7f805ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816092462 0 0) 0x7f807806f5e0 con 0x7f8078149dc0 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8064004eb0 con 0x7f8078149dc0 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 client.?.objecter ms_handle_connect 0x7f807814ff40 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797302326 0 0) 0x7f8078138960 con 0x7f807814ff40 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:13.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807806f5e0 con 0x7f807814ff40 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885474779 0 0) 0x7f8064004eb0 con 0x7f8078149dc0 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.053+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8078138960 con 0x7f8078149dc0 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 client.?.objecter ms_handle_connect 0x7f80780a9930 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8070001ef0 con 0x7f8078149dc0 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.048 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157237490 0 0) 0x7f807806f5e0 con 0x7f807814ff40 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8064004eb0 con 0x7f807814ff40 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f80680016c0 con 0x7f807814ff40 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:13.049 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844887382 0 0) 0x7f80780b2e90 con 0x7f80780a9930 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:41:13.050 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807806f5e0 con 0x7f80780a9930 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1385426631 0 0) 0x7f8064004eb0 con 0x7f807814ff40 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 >> v1:172.21.15.169:6789/0 conn(0x7f80780a9930 legacy=0x7f807813fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 >> v1:172.21.15.151:6789/0 conn(0x7f8078149dc0 legacy=0x7f807813d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 monclient: found mon.c 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807813dc80 con 0x7f807814ff40 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f805ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 10 monclient: _renew_subs 2024-03-29T10:41:13.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807813c940 con 0x7f807814ff40 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 10 monclient: _renew_subs 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.054+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f807813c130 con 0x7f807814ff40 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.055+0000 7f807e0d9640 1 librados: init done 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.055+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8068001c30 con 0x7f807814ff40 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.055+0000 7f805ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.055+0000 7f8075ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.056+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f80680037b0 con 0x7f807814ff40 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 10 monclient: dump: 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:13.052 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:13.053 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.053 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:13.053 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f8068003990 con 0x7f807814ff40 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f8068083600 con 0x7f807814ff40 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 10 client.4373.objecter ms_dispatch 0x7f8078000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:13.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.057+0000 7f805ffff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f80777fe640 10 monclient: get_auth_request con 0x7f806406dfe0 auth_method 0 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f805ffff640 10 client.4373.objecter ms_handle_connect 0x7f806406dfe0 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f807e0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f807e0d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f807e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.058+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8058005080 con 0x7f807814ff40 2024-03-29T10:41:13.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.066+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f806804efa0 con 0x7f807814ff40 2024-03-29T10:41:13.056 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.066+0000 7f805ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:13.056 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.066+0000 7f805ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:13.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.401+0000 7f807e0d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.401+0000 7f807e0d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.401+0000 7f807e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.401+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8058005330 con 0x7f807814ff40 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.402+0000 7f805ffff640 1 -- 172.21.15.151:0/2556939424 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f806804efa0 con 0x7f807814ff40 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.402+0000 7f805ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.402+0000 7f805ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:13.388 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:13.388 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:13.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.409+0000 7f807e0d9640 10 librados: watch_flush enter 2024-03-29T10:41:13.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.409+0000 7f807e0d9640 10 librados: watch_flush exit 2024-03-29T10:41:13.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.409+0000 7f807e0d9640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T10:41:13.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.409+0000 7f807e0d9640 10 client.4373.objecter successfully canceled tick 2024-03-29T10:41:13.394 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.410+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 >> v1:172.21.15.151:6816/1549873000 conn(0x7f806406dfe0 legacy=0x7f80640704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.394 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.410+0000 7f807e0d9640 10 monclient: shutdown 2024-03-29T10:41:13.394 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.410+0000 7f807e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:13.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.410+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 >> v1:172.21.15.151:6790/0 conn(0x7f807814ff40 legacy=0x7f80782000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.411+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 shutdown_connections 2024-03-29T10:41:13.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.411+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 >> 172.21.15.151:0/2556939424 conn(0x7f80780b35e0 msgr2=0x7f807813a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:13.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.413+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 shutdown_connections 2024-03-29T10:41:13.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.414+0000 7f807e0d9640 1 -- 172.21.15.151:0/2556939424 wait complete. 2024-03-29T10:41:13.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:13.414+0000 7f807e0d9640 1 librados: shutdown 2024-03-29T10:41:13.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:20.222 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:20.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.398+0000 7fa08d04d640 10 monclient: get_monmap_and_config 2024-03-29T10:41:20.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.398+0000 7fa08d04d640 10 monclient: build_initial_monmap 2024-03-29T10:41:20.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.398+0000 7fa08d04d640 10 monclient: monmap: 2024-03-29T10:41:20.382 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:20.399496+0000 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:20.399496+0000 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.400+0000 7fa08d04d640 1 Processor -- start 2024-03-29T10:41:20.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.400+0000 7fa08d04d640 1 -- start start 2024-03-29T10:41:20.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.400+0000 7fa08d04d640 10 monclient: init 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-c con 0x7fa080006890 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-a con 0x7fa080006c70 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-b con 0x7fa08002b4c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08002c830 con 0x7fa080006c70 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08002afa0 con 0x7fa080006890 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08002aba0 con 0x7fa08002b4c0 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa08d04d640 10 monclient(hunting): authenticate will time out at 1211.570681s 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa087fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa080006890 0x7fa0800c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:39928/0 (socket says 172.21.15.151:39928) 2024-03-29T10:41:20.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa087fff640 1 -- 172.21.15.151:0/2930985224 learned_addr learned my addr 172.21.15.151:0/2930985224 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853040216 0 0) 0x7fa08002afa0 con 0x7fa080006890 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:41:20.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.401+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078003520 con 0x7fa080006890 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852853567 0 0) 0x7fa08002aba0 con 0x7fa08002b4c0 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa08002afa0 con 0x7fa08002b4c0 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390848864 0 0) 0x7fa078003520 con 0x7fa080006890 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 >> v1:172.21.15.151:6790/0 conn(0x7fa08002b4c0 legacy=0x7fa0800c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 >> v1:172.21.15.151:6789/0 conn(0x7fa080006c70 legacy=0x7fa0800cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:20.388 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.402+0000 7fa086ffd640 1 monclient: found mon.noname-c 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08002ad30 con 0x7fa080006890 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa07c001610 con 0x7fa080006890 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa08d04d640 5 monclient: authenticate success, global_id 4387 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa08d04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:20.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: dump: 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa08d04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c001830 con 0x7fa080006890 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 1 -- 172.21.15.151:0/2930985224 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa07c001bc0 con 0x7fa080006890 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa086ffd640 10 monclient: dump: 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:20.391 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa08d04d640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.403+0000 7fa08d04d640 4 set_mon_vals no callback set 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.404+0000 7fa08d04d640 10 monclient: shutdown 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.404+0000 7fa08d04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:20.392 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.404+0000 7fa08d04d640 1 -- 172.21.15.151:0/2930985224 >> v1:172.21.15.169:6789/0 conn(0x7fa080006890 legacy=0x7fa0800c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.405+0000 7fa08d04d640 1 -- 172.21.15.151:0/2930985224 shutdown_connections 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.405+0000 7fa08d04d640 1 -- 172.21.15.151:0/2930985224 >> 172.21.15.151:0/2930985224 conn(0x7fa0800058f0 msgr2=0x7fa080005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.405+0000 7fa08d04d640 1 -- 172.21.15.151:0/2930985224 shutdown_connections 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.405+0000 7fa08d04d640 1 -- 172.21.15.151:0/2930985224 wait complete. 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.406+0000 7fa08d04d640 10 monclient: build_initial_monmap 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.406+0000 7fa08d04d640 10 monclient: monmap: 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:20.397 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.406+0000 7fa08d04d640 1 librados: starting msgr at 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.406+0000 7fa08d04d640 1 librados: starting objecter 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.406+0000 7fa08d04d640 1 Processor -- start 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.407+0000 7fa08d04d640 1 -- start start 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.407+0000 7fa08d04d640 1 librados: setting wanted keys 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.407+0000 7fa08d04d640 1 librados: calling monclient init 2024-03-29T10:41:20.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.407+0000 7fa08d04d640 10 monclient: init 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-b con 0x7fa080006890 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-c con 0x7fa08002b4c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient(hunting): picked mon.noname-a con 0x7fa080161b60 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800b57f0 con 0x7fa080161b60 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa080161640 con 0x7fa080006890 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800d60f0 con 0x7fa08002b4c0 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa08d04d640 10 monclient(hunting): authenticate will time out at 1211.577629s 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa087fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa080006890 0x7fa0800d59e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:40098/0 (socket says 172.21.15.151:40098) 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa087fff640 1 -- 172.21.15.151:0/992433314 learned_addr learned my addr 172.21.15.151:0/992433314 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 client.?.objecter ms_handle_connect 0x7fa080006890 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:20.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 client.?.objecter ms_handle_connect 0x7fa080161b60 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310671150 0 0) 0x7fa0800b57f0 con 0x7fa080161b60 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06c004eb0 con 0x7fa080161b60 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509430047 0 0) 0x7fa080161640 con 0x7fa080006890 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.408+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0800b57f0 con 0x7fa080006890 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 client.?.objecter ms_handle_connect 0x7fa08002b4c0 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718639405 0 0) 0x7fa0800b57f0 con 0x7fa080006890 2024-03-29T10:41:20.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa080161640 con 0x7fa080006890 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa07c001b80 con 0x7fa080006890 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:20.401 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 171950240 0 0) 0x7fa06c004eb0 con 0x7fa080161b60 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0800b57f0 con 0x7fa080161b60 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa08804fdd0 con 0x7fa080161b60 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:20.402 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265826454 0 0) 0x7fa0800d60f0 con 0x7fa08002b4c0 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.409+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06c004eb0 con 0x7fa08002b4c0 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1487950752 0 0) 0x7fa080161640 con 0x7fa080006890 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 >> v1:172.21.15.151:6790/0 conn(0x7fa08002b4c0 legacy=0x7fa0800b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 >> v1:172.21.15.151:6789/0 conn(0x7fa080161b60 legacy=0x7fa0800b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 monclient: found mon.b 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080031410 con 0x7fa080006890 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 10 monclient: _renew_subs 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa080031660 con 0x7fa080006890 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 10 monclient: _renew_subs 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0800c4020 con 0x7fa080006890 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa08d04d640 1 librados: init done 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c004a90 con 0x7fa080006890 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa07c001270 con 0x7fa080006890 2024-03-29T10:41:20.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.410+0000 7fa084ff9640 10 monclient: dump: 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.411+0000 7fa0867fc640 4 set_mon_vals no callback set 2024-03-29T10:41:20.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.411+0000 7fa08d04d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.412+0000 7fa08d04d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.412+0000 7fa08d04d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.412+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa068005080 con 0x7fa080006890 2024-03-29T10:41:20.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.413+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fa07c004ea0 con 0x7fa080006890 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fa07c084bf0 con 0x7fa080006890 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 10 client.4390.objecter ms_dispatch 0x7fa088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa0877fe640 10 monclient: get_auth_request con 0x7fa06c06de80 auth_method 0 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.414+0000 7fa084ff9640 10 client.4390.objecter ms_handle_connect 0x7fa06c06de80 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.419+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa07c0504d0 con 0x7fa080006890 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.420+0000 7fa084ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.420+0000 7fa084ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:20.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.779+0000 7fa08d04d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.779+0000 7fa08d04d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.779+0000 7fa08d04d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:41:20.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.779+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa068005330 con 0x7fa080006890 2024-03-29T10:41:20.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.780+0000 7fa084ff9640 1 -- 172.21.15.151:0/992433314 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fa07c052400 con 0x7fa080006890 2024-03-29T10:41:20.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.780+0000 7fa084ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.780+0000 7fa084ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:20.764 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:20.764 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:20.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 10 librados: watch_flush enter 2024-03-29T10:41:20.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 10 librados: watch_flush exit 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 10 client.4390.objecter successfully canceled tick 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 >> v1:172.21.15.151:6816/1549873000 conn(0x7fa06c06de80 legacy=0x7fa06c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 10 monclient: shutdown 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:20.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 >> v1:172.21.15.169:6789/0 conn(0x7fa080006890 legacy=0x7fa0800d59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 shutdown_connections 2024-03-29T10:41:20.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.784+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 >> 172.21.15.151:0/992433314 conn(0x7fa0800058f0 msgr2=0x7fa080038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:20.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.785+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 shutdown_connections 2024-03-29T10:41:20.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.785+0000 7fa08d04d640 1 -- 172.21.15.151:0/992433314 wait complete. 2024-03-29T10:41:20.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:20.785+0000 7fa08d04d640 1 librados: shutdown 2024-03-29T10:41:20.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:27.591 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.759+0000 7f9565225640 10 monclient: get_monmap_and_config 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.759+0000 7f9565225640 10 monclient: build_initial_monmap 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.759+0000 7f9565225640 10 monclient: monmap: 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:27.760314+0000 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:27.760314+0000 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:27.743 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:27.744 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:27.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.760+0000 7f9565225640 1 Processor -- start 2024-03-29T10:41:27.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 1 -- start start 2024-03-29T10:41:27.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient: init 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient(hunting): picked mon.noname-b con 0x7f95601457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient(hunting): picked mon.noname-a con 0x7f956013f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient(hunting): picked mon.noname-c con 0x7f95600a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:27.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600aaef0 con 0x7f956013f650 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560145bd0 con 0x7f95600a9d70 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560145eb0 con 0x7f95601457f0 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.761+0000 7f9565225640 10 monclient(hunting): authenticate will time out at 1218.931500s 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955ed9d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f95601457f0 0x7f956013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:58592/0 (socket says 172.21.15.151:58592) 2024-03-29T10:41:27.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955ed9d640 1 -- 172.21.15.151:0/366311255 learned_addr learned my addr 172.21.15.151:0/366311255 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284794394 0 0) 0x7f9560145eb0 con 0x7f95601457f0 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9548003520 con 0x7f95601457f0 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169899616 0 0) 0x7f95600aaef0 con 0x7f956013f650 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.762+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9560145eb0 con 0x7f956013f650 2024-03-29T10:41:27.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2182453674 0 0) 0x7f9548003520 con 0x7f95601457f0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 >> v1:172.21.15.169:6789/0 conn(0x7f95600a9d70 legacy=0x7f956014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 >> v1:172.21.15.151:6789/0 conn(0x7f956013f650 legacy=0x7f95600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 monclient: found mon.noname-b 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560146340 con 0x7f95601457f0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f9565225640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f9565225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9558001a10 con 0x7f95601457f0 2024-03-29T10:41:27.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: dump: 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:27.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f9565225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:27.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 1 -- 172.21.15.151:0/366311255 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9558001c30 con 0x7f95601457f0 2024-03-29T10:41:27.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f955dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:27.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f9565225640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.763+0000 7f9565225640 4 set_mon_vals no callback set 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 10 monclient: shutdown 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 1 -- 172.21.15.151:0/366311255 >> v1:172.21.15.151:6790/0 conn(0x7f95601457f0 legacy=0x7f956013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 1 -- 172.21.15.151:0/366311255 shutdown_connections 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 1 -- 172.21.15.151:0/366311255 >> 172.21.15.151:0/366311255 conn(0x7f9560066a00 msgr2=0x7f9560066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 1 -- 172.21.15.151:0/366311255 shutdown_connections 2024-03-29T10:41:27.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.764+0000 7f9565225640 1 -- 172.21.15.151:0/366311255 wait complete. 2024-03-29T10:41:27.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.765+0000 7f9565225640 10 monclient: build_initial_monmap 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.765+0000 7f9565225640 10 monclient: monmap: 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.765+0000 7f9565225640 1 librados: starting msgr at 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.765+0000 7f9565225640 1 librados: starting objecter 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 1 Processor -- start 2024-03-29T10:41:27.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 1 -- start start 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 1 librados: setting wanted keys 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 1 librados: calling monclient init 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient: init 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient(hunting): picked mon.noname-b con 0x7f95600a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient(hunting): picked mon.noname-a con 0x7f956013f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.766+0000 7f9565225640 10 monclient(hunting): picked mon.noname-c con 0x7f95601457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f955f59e640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f95601457f0 0x7f95601e1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:58934/0 (socket says 172.21.15.151:58934) 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f955f59e640 1 -- 172.21.15.151:0/648301173 learned_addr learned my addr 172.21.15.151:0/648301173 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560140b00 con 0x7f956013f650 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560141b60 con 0x7f95600a9d70 2024-03-29T10:41:27.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter ms_handle_connect 0x7f95601457f0 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560141dd0 con 0x7f95601457f0 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f9565225640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f9565225640 10 monclient(hunting): authenticate will time out at 1218.936826s 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68550883 0 0) 0x7f9560141dd0 con 0x7f95601457f0 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c004eb0 con 0x7f95601457f0 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter ms_handle_connect 0x7f95600a9d70 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter ms_handle_connect 0x7f956013f650 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.767+0000 7f95477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:27.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1692012990 0 0) 0x7f954c004eb0 con 0x7f95601457f0 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9560141dd0 con 0x7f95601457f0 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9554004000 con 0x7f95601457f0 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): dump: 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:27.760 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339901271 0 0) 0x7f9560141b60 con 0x7f95600a9d70 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c004eb0 con 0x7f95600a9d70 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180576877 0 0) 0x7f9560141dd0 con 0x7f95601457f0 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 >> v1:172.21.15.169:6789/0 conn(0x7f95600a9d70 legacy=0x7f956013b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 >> v1:172.21.15.151:6789/0 conn(0x7f956013f650 legacy=0x7f95601403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 monclient: found mon.c 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:27.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956013fed0 con 0x7f95601457f0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 10 monclient: _renew_subs 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9560142560 con 0x7f95601457f0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 10 monclient: _renew_subs 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95601e3130 con 0x7f95601457f0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f9565225640 1 librados: init done 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9554000ed0 con 0x7f95601457f0 2024-03-29T10:41:27.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f95540012e0 con 0x7f95601457f0 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f95477fe640 10 monclient: dump: 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:27.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.768+0000 7f955cd99640 4 set_mon_vals no callback set 2024-03-29T10:41:27.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.770+0000 7f9565225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.770+0000 7f9565225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.770+0000 7f9565225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:27.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.770+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540005080 con 0x7f95601457f0 2024-03-29T10:41:27.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.771+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f9554001440 con 0x7f95601457f0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.773+0000 7f955ed9d640 10 monclient: get_auth_request con 0x7f954c06ddb0 auth_method 0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f95540849c0 con 0x7f95601457f0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 10 client.4385.objecter ms_dispatch 0x7f9560000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.774+0000 7f95477fe640 10 client.4385.objecter ms_handle_connect 0x7f954c06ddb0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.781+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9554050300 con 0x7f95601457f0 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.781+0000 7f95477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:27.781+0000 7f95477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:28.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.097+0000 7f9565225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:28.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.097+0000 7f9565225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:28.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.097+0000 7f9565225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:28.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.097+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9540005330 con 0x7f95601457f0 2024-03-29T10:41:28.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.098+0000 7f95477fe640 1 -- 172.21.15.151:0/648301173 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f9554052230 con 0x7f95601457f0 2024-03-29T10:41:28.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.098+0000 7f95477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:28.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.098+0000 7f95477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:28.083 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:28.083 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:28.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.102+0000 7f9565225640 10 librados: watch_flush enter 2024-03-29T10:41:28.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 10 librados: watch_flush exit 2024-03-29T10:41:28.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T10:41:28.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 10 client.4385.objecter successfully canceled tick 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 >> v1:172.21.15.151:6816/1549873000 conn(0x7f954c06ddb0 legacy=0x7f954c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 10 monclient: shutdown 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 >> v1:172.21.15.151:6790/0 conn(0x7f95601457f0 legacy=0x7f95601e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 shutdown_connections 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 >> 172.21.15.151:0/648301173 conn(0x7f9560066a00 msgr2=0x7f95600b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 shutdown_connections 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 -- 172.21.15.151:0/648301173 wait complete. 2024-03-29T10:41:28.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:28.103+0000 7f9565225640 1 librados: shutdown 2024-03-29T10:41:28.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:34.910 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.078+0000 7f4c1c132640 10 monclient: get_monmap_and_config 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.078+0000 7f4c1c132640 10 monclient: build_initial_monmap 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.078+0000 7f4c1c132640 10 monclient: monmap: 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:35.079031+0000 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:35.079031+0000 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.063 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:35.064 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:35.064 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:35.064 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.080+0000 7f4c1c132640 1 Processor -- start 2024-03-29T10:41:35.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.080+0000 7f4c1c132640 1 -- start start 2024-03-29T10:41:35.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.080+0000 7f4c1c132640 10 monclient: init 2024-03-29T10:41:35.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.081+0000 7f4c1c132640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-c con 0x7f4c140a9ac0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-a con 0x7f4c1413ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-b con 0x7f4c140af900 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140afd10 con 0x7f4c1413ee90 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140af6f0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140aac80 con 0x7f4c140af900 2024-03-29T10:41:35.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:35.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.082+0000 7f4c1c132640 10 monclient(hunting): authenticate will time out at 1226.252261s 2024-03-29T10:41:35.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.085+0000 7f4c19ece640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4c140a9ac0 0x7f4c140adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:50387/0 (socket says 172.21.15.151:50387) 2024-03-29T10:41:35.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.085+0000 7f4c19ece640 1 -- 172.21.15.151:0/3411882748 learned_addr learned my addr 172.21.15.151:0/3411882748 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:35.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.087+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929502178 0 0) 0x7f4c140aac80 con 0x7f4c140af900 2024-03-29T10:41:35.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:35.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:35.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:35.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c00003520 con 0x7f4c140af900 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.088+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851327153 0 0) 0x7f4c140af6f0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:41:35.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c140aac80 con 0x7f4c140a9ac0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3526894818 0 0) 0x7f4c140aac80 con 0x7f4c140a9ac0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 >> v1:172.21.15.151:6790/0 conn(0x7f4c140af900 legacy=0x7f4c14149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 >> v1:172.21.15.151:6789/0 conn(0x7f4c1413ee90 legacy=0x7f4c1413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 monclient: found mon.noname-c 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.089+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140aaf60 con 0x7f4c140a9ac0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: _finish_auth 0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c0c0019f0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c1c132640 5 monclient: authenticate success, global_id 4396 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c1c132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:35.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: dump: 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c1c132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 1 -- 172.21.15.151:0/3411882748 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c0c001c40 con 0x7f4c140a9ac0 2024-03-29T10:41:35.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c18ecc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:35.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c1c132640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:35.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.090+0000 7f4c1c132640 4 set_mon_vals no callback set 2024-03-29T10:41:35.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.091+0000 7f4c1c132640 10 monclient: shutdown 2024-03-29T10:41:35.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.091+0000 7f4c1c132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:35.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.091+0000 7f4c1c132640 1 -- 172.21.15.151:0/3411882748 >> v1:172.21.15.169:6789/0 conn(0x7f4c140a9ac0 legacy=0x7f4c140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.092+0000 7f4c1c132640 1 -- 172.21.15.151:0/3411882748 shutdown_connections 2024-03-29T10:41:35.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.092+0000 7f4c1c132640 1 -- 172.21.15.151:0/3411882748 >> 172.21.15.151:0/3411882748 conn(0x7f4c140b35e0 msgr2=0x7f4c140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.093+0000 7f4c1c132640 1 -- 172.21.15.151:0/3411882748 shutdown_connections 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.093+0000 7f4c1c132640 1 -- 172.21.15.151:0/3411882748 wait complete. 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.093+0000 7f4c1c132640 10 monclient: build_initial_monmap 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.093+0000 7f4c1c132640 10 monclient: monmap: 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 librados: starting msgr at 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 librados: starting objecter 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 Processor -- start 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 -- start start 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 librados: setting wanted keys 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 1 librados: calling monclient init 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.094+0000 7f4c1c132640 10 monclient: init 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-a con 0x7f4c140a9ac0 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-b con 0x7f4c140af900 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient(hunting): picked mon.noname-c con 0x7f4c1413ee90 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c141e3ed0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c141e4120 con 0x7f4c140af900 2024-03-29T10:41:35.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c141e4d30 con 0x7f4c1413ee90 2024-03-29T10:41:35.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:35.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.095+0000 7f4c1c132640 10 monclient(hunting): authenticate will time out at 1226.265105s 2024-03-29T10:41:35.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.098+0000 7f4c1a6cf640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f4c1413ee90 0x7f4c141ee930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:36683/0 (socket says 172.21.15.151:36683) 2024-03-29T10:41:35.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.098+0000 7f4c1a6cf640 1 -- 172.21.15.151:0/1632378619 learned_addr learned my addr 172.21.15.151:0/1632378619 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:35.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c1413ee90 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c140a9ac0 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f4c140af900 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426325387 0 0) 0x7f4c141e4d30 con 0x7f4c1413ee90 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.102+0000 7f4c0a7fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8004eb0 con 0x7f4c1413ee90 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211162679 0 0) 0x7f4c141e3ed0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c141e4d30 con 0x7f4c140a9ac0 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954309580 0 0) 0x7f4c141e4120 con 0x7f4c140af900 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c141e3ed0 con 0x7f4c140af900 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.103+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536440937 0 0) 0x7f4bf8004eb0 con 0x7f4c1413ee90 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c141e4120 con 0x7f4c1413ee90 2024-03-29T10:41:35.092 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1880049607 0 0) 0x7f4c141e4d30 con 0x7f4c140a9ac0 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bf8004eb0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c100018d0 con 0x7f4c1413ee90 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:35.093 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c0c0019b0 con 0x7f4c140a9ac0 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:35.094 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3750982551 0 0) 0x7f4c141e4120 con 0x7f4c1413ee90 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 >> v1:172.21.15.169:6789/0 conn(0x7f4c140af900 legacy=0x7f4c141c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 >> v1:172.21.15.151:6789/0 conn(0x7f4c140a9ac0 legacy=0x7f4c141e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.104+0000 7f4c0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 1 monclient: found mon.c 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c141c4410 con 0x7f4c1413ee90 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:35.095 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 5 monclient: authenticate success, global_id 4391 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 10 monclient: _renew_subs 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1406ff60 con 0x7f4c1413ee90 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 10 monclient: _renew_subs 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c141c5570 con 0x7f4c1413ee90 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c1c132640 1 librados: init done 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c10003610 con 0x7f4c1413ee90 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.105+0000 7f4c0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:35.096 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.106+0000 7f4c0b7fe640 4 set_mon_vals no callback set 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c100039e0 con 0x7f4c1413ee90 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c0a7fc640 10 monclient: dump: 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:35.098 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4c10003c00 con 0x7f4c1413ee90 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c1c132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c1c132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c1c132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.108+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc005080 con 0x7f4c1413ee90 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.109+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4c10084940 con 0x7f4c1413ee90 2024-03-29T10:41:35.099 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.109+0000 7f4c19ece640 10 monclient: get_auth_request con 0x7f4bf806e210 auth_method 0 2024-03-29T10:41:35.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.110+0000 7f4c0a7fc640 10 client.4391.objecter ms_dispatch 0x7f4c14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:35.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.110+0000 7f4c0a7fc640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:35.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.110+0000 7f4c0a7fc640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:35.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.110+0000 7f4c0a7fc640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T10:41:35.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.110+0000 7f4c0a7fc640 10 client.4391.objecter ms_handle_connect 0x7f4bf806e210 2024-03-29T10:41:35.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.119+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c100502a0 con 0x7f4c1413ee90 2024-03-29T10:41:35.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.119+0000 7f4c0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:35.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.119+0000 7f4c0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:35.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.435+0000 7f4c1c132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.435+0000 7f4c1c132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.435+0000 7f4c1c132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:35.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.435+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bfc005330 con 0x7f4c1413ee90 2024-03-29T10:41:35.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.437+0000 7f4c0a7fc640 1 -- 172.21.15.151:0/1632378619 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4c100521d0 con 0x7f4c1413ee90 2024-03-29T10:41:35.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.437+0000 7f4c0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.437+0000 7f4c0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:35.422 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:35.422 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 10 librados: watch_flush enter 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 10 librados: watch_flush exit 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 10 client.4391.objecter successfully canceled tick 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4bf806e210 legacy=0x7f4bf80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 10 monclient: shutdown 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.441+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 >> v1:172.21.15.151:6790/0 conn(0x7f4c1413ee90 legacy=0x7f4c141ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.442+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 shutdown_connections 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.442+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 >> 172.21.15.151:0/1632378619 conn(0x7f4c140b35e0 msgr2=0x7f4c140ad270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.443+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 shutdown_connections 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.443+0000 7f4c1c132640 1 -- 172.21.15.151:0/1632378619 wait complete. 2024-03-29T10:41:35.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:35.443+0000 7f4c1c132640 1 librados: shutdown 2024-03-29T10:41:35.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:42.255 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.421+0000 7f12b72a8640 10 monclient: get_monmap_and_config 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.421+0000 7f12b72a8640 10 monclient: build_initial_monmap 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.421+0000 7f12b72a8640 10 monclient: monmap: 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:42.405 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:42.422723+0000 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:42.422723+0000 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.423+0000 7f12b72a8640 1 Processor -- start 2024-03-29T10:41:42.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.423+0000 7f12b72a8640 1 -- start start 2024-03-29T10:41:42.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.423+0000 7f12b72a8640 10 monclient: init 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-b con 0x7f12b00b01a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-a con 0x7f12b00af010 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-c con 0x7f12b00ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00ae8a0 con 0x7f12b00af010 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00ab1d0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00ab4b0 con 0x7f12b00b01a0 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:42.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.424+0000 7f12b72a8640 10 monclient(hunting): authenticate will time out at 1233.593840s 2024-03-29T10:41:42.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.426+0000 7f12b5044640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f12b00b01a0 0x7f12b00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:58684/0 (socket says 172.21.15.151:58684) 2024-03-29T10:41:42.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.426+0000 7f12b5044640 1 -- 172.21.15.151:0/3845748708 learned_addr learned my addr 172.21.15.151:0/3845748708 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.426+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816671518 0 0) 0x7f12b00ab4b0 con 0x7f12b00b01a0 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.427+0000 7f12a7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.427+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.427+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.427+0000 7f12a7fff640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:41:42.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.427+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c003520 con 0x7f12b00b01a0 2024-03-29T10:41:42.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159291073 0 0) 0x7f12b00ab1d0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00ab4b0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370398598 0 0) 0x7f12b00ae8a0 con 0x7f12b00af010 2024-03-29T10:41:42.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.428+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00ab1d0 con 0x7f12b00af010 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905520994 0 0) 0x7f12b00ab4b0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 >> v1:172.21.15.151:6790/0 conn(0x7f12b00b01a0 legacy=0x7f12b00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 >> v1:172.21.15.151:6789/0 conn(0x7f12b00af010 legacy=0x7f12b0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 monclient: found mon.noname-c 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00ab790 con 0x7f12b00ac5a0 2024-03-29T10:41:42.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12ac001670 con 0x7f12b00ac5a0 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: dump: 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:42.416 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12b72a8640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12b72a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12ac0017d0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 1 -- 172.21.15.151:0/3845748708 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12ac001be0 con 0x7f12b00ac5a0 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12a7fff640 10 monclient: dump: 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.417 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12b72a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.429+0000 7f12b72a8640 4 set_mon_vals no callback set 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.430+0000 7f12b72a8640 10 monclient: shutdown 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.430+0000 7f12b72a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.430+0000 7f12b72a8640 1 -- 172.21.15.151:0/3845748708 >> v1:172.21.15.169:6789/0 conn(0x7f12b00ac5a0 legacy=0x7f12b0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.430+0000 7f12b72a8640 1 -- 172.21.15.151:0/3845748708 shutdown_connections 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.430+0000 7f12b72a8640 1 -- 172.21.15.151:0/3845748708 >> 172.21.15.151:0/3845748708 conn(0x7f12b006b660 msgr2=0x7f12b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.431+0000 7f12b72a8640 1 -- 172.21.15.151:0/3845748708 shutdown_connections 2024-03-29T10:41:42.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.431+0000 7f12b72a8640 1 -- 172.21.15.151:0/3845748708 wait complete. 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.432+0000 7f12b72a8640 10 monclient: build_initial_monmap 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.432+0000 7f12b72a8640 10 monclient: monmap: 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:42.422 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.433+0000 7f12b72a8640 1 librados: starting msgr at 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.433+0000 7f12b72a8640 1 librados: starting objecter 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.433+0000 7f12b72a8640 1 Processor -- start 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.433+0000 7f12b72a8640 1 -- start start 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.433+0000 7f12b72a8640 1 librados: setting wanted keys 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.434+0000 7f12b72a8640 1 librados: calling monclient init 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.434+0000 7f12b72a8640 10 monclient: init 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-b con 0x7f12b00ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-a con 0x7f12b00b01a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient(hunting): picked mon.noname-c con 0x7f12b0145770 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00702d0 con 0x7f12b00b01a0 2024-03-29T10:41:42.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b0070460 con 0x7f12b00ac5a0 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00705f0 con 0x7f12b0145770 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.435+0000 7f12b72a8640 10 monclient(hunting): authenticate will time out at 1233.604956s 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.436+0000 7f12b5845640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f12b0145770 0x7f12b01df0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60452/0 (socket says 172.21.15.151:60452) 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.436+0000 7f12b5845640 1 -- 172.21.15.151:0/596601495 learned_addr learned my addr 172.21.15.151:0/596601495 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 client.?.objecter ms_handle_connect 0x7f12b0145770 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859763423 0 0) 0x7f12b00705f0 con 0x7f12b0145770 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1294004eb0 con 0x7f12b0145770 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.437+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4191007378 0 0) 0x7f1294004eb0 con 0x7f12b0145770 2024-03-29T10:41:42.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12b00705f0 con 0x7f12b0145770 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12ac001be0 con 0x7f12b0145770 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 10 monclient(hunting): dump: 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:42.425 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:42.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 920277888 0 0) 0x7f12b00705f0 con 0x7f12b0145770 2024-03-29T10:41:42.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.438+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 >> v1:172.21.15.169:6789/0 conn(0x7f12b00ac5a0 legacy=0x7f12b01db830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:42.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 >> v1:172.21.15.151:6789/0 conn(0x7f12b00b01a0 legacy=0x7f12b01de990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:42.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 1 monclient: found mon.c 2024-03-29T10:41:42.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b01e9f00 con 0x7f12b0145770 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.439+0000 7f12a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12b72a8640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12b72a8640 10 monclient: _renew_subs 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12b72a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12b72a8640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b01e6ed0 con 0x7f12b0145770 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12ac0038a0 con 0x7f12b0145770 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f12ac003cf0 con 0x7f12b0145770 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.440+0000 7f12a5ffb640 10 monclient: dump: 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.442+0000 7f12b72a8640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.442+0000 7f12b72a8640 10 monclient: _renew_subs 2024-03-29T10:41:42.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.442+0000 7f12b72a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.442+0000 7f12b72a8640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b01e7350 con 0x7f12b0145770 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.442+0000 7f12b72a8640 1 librados: init done 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.443+0000 7f12a77fe640 4 set_mon_vals no callback set 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.445+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f12ac003e50 con 0x7f12b0145770 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.445+0000 7f129b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.445+0000 7f129b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.445+0000 7f129b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.445+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b0075f10 con 0x7f12b0145770 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.446+0000 7f12b5044640 10 monclient: get_auth_request con 0x7f129406dbc0 auth_method 0 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 10 client.4397.objecter ms_handle_connect 0x7f129406dbc0 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f12ac054ab0 con 0x7f12b0145770 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 10 client.4397.objecter ms_dispatch 0x7f12b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:42.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.447+0000 7f12a5ffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:41:42.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.455+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12ac0505c0 con 0x7f12b0145770 2024-03-29T10:41:42.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.455+0000 7f12a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.455+0000 7f12a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:42.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.778+0000 7f129b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.778+0000 7f129b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.778+0000 7f129b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:42.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.778+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12b01dc170 con 0x7f12b0145770 2024-03-29T10:41:42.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.779+0000 7f12a5ffb640 1 -- 172.21.15.151:0/596601495 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f12ac084330 con 0x7f12b0145770 2024-03-29T10:41:42.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.779+0000 7f12a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.779+0000 7f12a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:42.764 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:42.764 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 10 librados: watch_flush enter 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 10 librados: watch_flush exit 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 >> v1:172.21.15.151:6816/1549873000 conn(0x7f129406dbc0 legacy=0x7f1294070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 10 monclient: shutdown 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 >> v1:172.21.15.151:6790/0 conn(0x7f12b0145770 legacy=0x7f12b01df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 shutdown_connections 2024-03-29T10:41:42.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.783+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 >> 172.21.15.151:0/596601495 conn(0x7f12b006b660 msgr2=0x7f12b006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:42.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.785+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 shutdown_connections 2024-03-29T10:41:42.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.785+0000 7f129b7fe640 1 -- 172.21.15.151:0/596601495 wait complete. 2024-03-29T10:41:42.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:42.786+0000 7f129b7fe640 1 librados: shutdown 2024-03-29T10:41:42.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:49.596 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.773+0000 7fe98fc5f640 10 monclient: get_monmap_and_config 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.773+0000 7fe98fc5f640 10 monclient: build_initial_monmap 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.773+0000 7fe98fc5f640 10 monclient: monmap: 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:49.774310+0000 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:49.774310+0000 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:49.757 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:49.758 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.775+0000 7fe98fc5f640 1 Processor -- start 2024-03-29T10:41:49.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.775+0000 7fe98fc5f640 1 -- start start 2024-03-29T10:41:49.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.775+0000 7fe98fc5f640 10 monclient: init 2024-03-29T10:41:49.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.776+0000 7fe98fc5f640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:49.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.776+0000 7fe98fc5f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:49.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.776+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe9881457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:49.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.776+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe98813f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:49.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.776+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe9880a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:49.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.779+0000 7fe98fc5f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9880aaef0 con 0x7fe98813f650 2024-03-29T10:41:49.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.779+0000 7fe98fc5f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988145bd0 con 0x7fe9881457f0 2024-03-29T10:41:49.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.779+0000 7fe98fc5f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988145eb0 con 0x7fe9880a9d70 2024-03-29T10:41:49.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.779+0000 7fe98fc5f640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:49.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.779+0000 7fe98fc5f640 10 monclient(hunting): authenticate will time out at 1240.948967s 2024-03-29T10:41:49.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.780+0000 7fe98d9fb640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe9881457f0 0x7fe98813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58223/0 (socket says 172.21.15.151:58223) 2024-03-29T10:41:49.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.780+0000 7fe98e1fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe9880a9d70 0x7fe98814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45737/0 (socket says 172.21.15.151:45737) 2024-03-29T10:41:49.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.780+0000 7fe98e1fc640 1 -- 172.21.15.151:0/1100436233 learned_addr learned my addr 172.21.15.151:0/1100436233 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183207948 0 0) 0x7fe988145bd0 con 0x7fe9881457f0 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe974003520 con 0x7fe9881457f0 2024-03-29T10:41:49.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649758828 0 0) 0x7fe974003520 con 0x7fe9881457f0 2024-03-29T10:41:49.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 >> v1:172.21.15.151:6790/0 conn(0x7fe9880a9d70 legacy=0x7fe98814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:49.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 >> v1:172.21.15.151:6789/0 conn(0x7fe98813f650 legacy=0x7fe9880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:49.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:49.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 monclient: found mon.noname-c 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988146340 con 0x7fe9881457f0 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: _finish_auth 0 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe980001a10 con 0x7fe9881457f0 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: dump: 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.767 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe980001ca0 con 0x7fe9881457f0 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 1 -- 172.21.15.151:0/1100436233 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe980002030 con 0x7fe9881457f0 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.781+0000 7fe98c9f9640 10 monclient: dump: 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.768 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.783+0000 7fe98fc5f640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.783+0000 7fe98fc5f640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:49.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.783+0000 7fe98fc5f640 4 set_mon_vals no callback set 2024-03-29T10:41:49.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.784+0000 7fe98fc5f640 10 monclient: shutdown 2024-03-29T10:41:49.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.784+0000 7fe98fc5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:49.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.784+0000 7fe98fc5f640 1 -- 172.21.15.151:0/1100436233 >> v1:172.21.15.169:6789/0 conn(0x7fe9881457f0 legacy=0x7fe98813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.785+0000 7fe98fc5f640 1 -- 172.21.15.151:0/1100436233 shutdown_connections 2024-03-29T10:41:49.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.785+0000 7fe98fc5f640 1 -- 172.21.15.151:0/1100436233 >> 172.21.15.151:0/1100436233 conn(0x7fe988066a00 msgr2=0x7fe988066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:49.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.786+0000 7fe98fc5f640 1 -- 172.21.15.151:0/1100436233 shutdown_connections 2024-03-29T10:41:49.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.786+0000 7fe98fc5f640 1 -- 172.21.15.151:0/1100436233 wait complete. 2024-03-29T10:41:49.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.787+0000 7fe98fc5f640 10 monclient: build_initial_monmap 2024-03-29T10:41:49.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.787+0000 7fe98fc5f640 10 monclient: monmap: 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 librados: starting msgr at 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 librados: starting objecter 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 Processor -- start 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 -- start start 2024-03-29T10:41:49.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 librados: setting wanted keys 2024-03-29T10:41:49.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 1 librados: calling monclient init 2024-03-29T10:41:49.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.788+0000 7fe98fc5f640 10 monclient: init 2024-03-29T10:41:49.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.789+0000 7fe98fc5f640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:49.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.789+0000 7fe98fc5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:49.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.789+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe9880a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:49.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.789+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe98813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.792+0000 7fe98fc5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe9881457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98fc5f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9881352a0 con 0x7fe9880a9d70 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98fc5f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988133e40 con 0x7fe98813f650 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98fc5f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9880b1730 con 0x7fe9881457f0 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98fc5f640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98fc5f640 10 monclient(hunting): authenticate will time out at 1240.962600s 2024-03-29T10:41:49.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.792+0000 7fe98d9fb640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe9880a9d70 0x7fe98812fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35261/0 (socket says 172.21.15.151:35261) 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98d9fb640 1 -- 172.21.15.151:0/2366132712 learned_addr learned my addr 172.21.15.151:0/2366132712 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe98d1fa640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe98813f650 0x7fe9881359a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58325/0 (socket says 172.21.15.151:58325) 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe97e7fc640 10 client.?.objecter ms_handle_connect 0x7fe9880a9d70 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.793+0000 7fe97e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 client.?.objecter ms_handle_connect 0x7fe98813f650 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641769603 0 0) 0x7fe9881352a0 con 0x7fe9880a9d70 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96c004eb0 con 0x7fe9880a9d70 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642407819 0 0) 0x7fe988133e40 con 0x7fe98813f650 2024-03-29T10:41:49.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.794+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9881352a0 con 0x7fe98813f650 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.795+0000 7fe97e7fc640 10 client.?.objecter ms_handle_connect 0x7fe9881457f0 2024-03-29T10:41:49.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.795+0000 7fe97e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238936141 0 0) 0x7fe96c004eb0 con 0x7fe9880a9d70 2024-03-29T10:41:49.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe988133e40 con 0x7fe9880a9d70 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2823951570 0 0) 0x7fe9881352a0 con 0x7fe98813f650 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe96c004eb0 con 0x7fe98813f650 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861568227 0 0) 0x7fe9880b1730 con 0x7fe9881457f0 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.798+0000 7fe97e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9881352a0 con 0x7fe9881457f0 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe9800019d0 con 0x7fe9880a9d70 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:49.785 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe984001760 con 0x7fe98813f650 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:49.786 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3133531988 0 0) 0x7fe988133e40 con 0x7fe9880a9d70 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 >> v1:172.21.15.151:6790/0 conn(0x7fe9881457f0 legacy=0x7fe9881360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 >> v1:172.21.15.169:6789/0 conn(0x7fe98813f650 legacy=0x7fe9881359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 monclient: found mon.a 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9880b18c0 con 0x7fe9880a9d70 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe97e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 5 monclient: authenticate success, global_id 4392 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 10 monclient: _renew_subs 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9881304d0 con 0x7fe9880a9d70 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 10 monclient: _renew_subs 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:41:49.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe988141aa0 con 0x7fe9880a9d70 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.799+0000 7fe98fc5f640 1 librados: init done 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9800015f0 con 0x7fe9880a9d70 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe980004e10 con 0x7fe9880a9d70 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97ffff640 4 set_mon_vals no callback set 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.800+0000 7fe97e7fc640 10 monclient: dump: 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:49.789 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:49.790 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:49.790 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:49.790 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.802+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe980004f70 con 0x7fe9880a9d70 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.803+0000 7fe98fc5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.803+0000 7fe98fc5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.803+0000 7fe98fc5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.803+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970005080 con 0x7fe9880a9d70 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe980084cb0 con 0x7fe9880a9d70 2024-03-29T10:41:49.791 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 10 client.4392.objecter ms_dispatch 0x7fe988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:49.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:49.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:49.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe98d1fa640 10 monclient: get_auth_request con 0x7fe96c06e1d0 auth_method 0 2024-03-29T10:41:49.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T10:41:49.792 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.804+0000 7fe97e7fc640 10 client.4392.objecter ms_handle_connect 0x7fe96c06e1d0 2024-03-29T10:41:49.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.811+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe980050590 con 0x7fe9880a9d70 2024-03-29T10:41:49.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.812+0000 7fe97e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.795 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:49.812+0000 7fe97e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:50.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.143+0000 7fe98fc5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:50.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.144+0000 7fe98fc5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:50.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.144+0000 7fe98fc5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:41:50.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.144+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe970005330 con 0x7fe9880a9d70 2024-03-29T10:41:50.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.145+0000 7fe97e7fc640 1 -- 172.21.15.151:0/2366132712 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe9800524c0 con 0x7fe9880a9d70 2024-03-29T10:41:50.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.145+0000 7fe97e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:50.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.145+0000 7fe97e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:50.130 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:50.130 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:50.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 10 librados: watch_flush enter 2024-03-29T10:41:50.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 10 librados: watch_flush exit 2024-03-29T10:41:50.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T10:41:50.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 10 client.4392.objecter successfully canceled tick 2024-03-29T10:41:50.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe96c06e1d0 legacy=0x7fe96c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 10 monclient: shutdown 2024-03-29T10:41:50.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:50.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.150+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 >> v1:172.21.15.151:6789/0 conn(0x7fe9880a9d70 legacy=0x7fe98812fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.151+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 shutdown_connections 2024-03-29T10:41:50.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.151+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 >> 172.21.15.151:0/2366132712 conn(0x7fe988066a00 msgr2=0x7fe9880b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:50.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.152+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 shutdown_connections 2024-03-29T10:41:50.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.152+0000 7fe98fc5f640 1 -- 172.21.15.151:0/2366132712 wait complete. 2024-03-29T10:41:50.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:50.152+0000 7fe98fc5f640 1 librados: shutdown 2024-03-29T10:41:50.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:56.960 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:57.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.135+0000 7f76baaae640 10 monclient: get_monmap_and_config 2024-03-29T10:41:57.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.135+0000 7f76baaae640 10 monclient: build_initial_monmap 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.135+0000 7f76baaae640 10 monclient: monmap: 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:41:57.136185+0000 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:41:57.136185+0000 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:41:57.120 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.137+0000 7f76baaae640 1 Processor -- start 2024-03-29T10:41:57.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.137+0000 7f76baaae640 1 -- start start 2024-03-29T10:41:57.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.137+0000 7f76baaae640 10 monclient: init 2024-03-29T10:41:57.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:57.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:57.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-b con 0x7f76b40aa2a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:57.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-a con 0x7f76b4141f20 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:57.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-c con 0x7f76b413b3c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:57.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b413e5c0 con 0x7f76b4141f20 2024-03-29T10:41:57.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40ab190 con 0x7f76b413b3c0 2024-03-29T10:41:57.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40ab470 con 0x7f76b40aa2a0 2024-03-29T10:41:57.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:57.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.138+0000 7f76baaae640 10 monclient(hunting): authenticate will time out at 1248.307892s 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b884a640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f76b40aa2a0 0x7f76b40ad270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:59573/0 (socket says 172.21.15.151:59573) 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b884a640 1 -- 172.21.15.151:0/1885657607 learned_addr learned my addr 172.21.15.151:0/1885657607 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200622497 0 0) 0x7f76b40ab470 con 0x7f76b40aa2a0 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.142+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f769c003520 con 0x7f76b40aa2a0 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149489985 0 0) 0x7f76b413e5c0 con 0x7f76b4141f20 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:57.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:41:57.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40ab470 con 0x7f76b4141f20 2024-03-29T10:41:57.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506430875 0 0) 0x7f769c003520 con 0x7f76b40aa2a0 2024-03-29T10:41:57.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.143+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 >> v1:172.21.15.169:6789/0 conn(0x7f76b413b3c0 legacy=0x7f76b413b810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.144+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 >> v1:172.21.15.151:6789/0 conn(0x7f76b4141f20 legacy=0x7f76b4142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 1 monclient: found mon.noname-b 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b4142a50 con 0x7f76b40aa2a0 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76baaae640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76baaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f76a80015e0 con 0x7f76b40aa2a0 2024-03-29T10:41:57.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76b37fe640 10 monclient: dump: 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.145+0000 7f76baaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.146+0000 7f76b37fe640 1 -- 172.21.15.151:0/1885657607 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a8001740 con 0x7f76b40aa2a0 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.146+0000 7f76b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.146+0000 7f76baaae640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.146+0000 7f76baaae640 4 set_mon_vals no callback set 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 10 monclient: shutdown 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 1 -- 172.21.15.151:0/1885657607 >> v1:172.21.15.151:6790/0 conn(0x7f76b40aa2a0 legacy=0x7f76b40ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 1 -- 172.21.15.151:0/1885657607 shutdown_connections 2024-03-29T10:41:57.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 1 -- 172.21.15.151:0/1885657607 >> 172.21.15.151:0/1885657607 conn(0x7f76b40b35e0 msgr2=0x7f76b40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:57.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 1 -- 172.21.15.151:0/1885657607 shutdown_connections 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.147+0000 7f76baaae640 1 -- 172.21.15.151:0/1885657607 wait complete. 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 10 monclient: build_initial_monmap 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 10 monclient: monmap: 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:57.133 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 librados: starting msgr at 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 librados: starting objecter 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 Processor -- start 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 -- start start 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 librados: setting wanted keys 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 1 librados: calling monclient init 2024-03-29T10:41:57.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.148+0000 7f76baaae640 10 monclient: init 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-b con 0x7f76b413b3c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-a con 0x7f76b4141f20 addr v1:172.21.15.151:6789/0 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient(hunting): picked mon.noname-c con 0x7f76b4137360 addr v1:172.21.15.151:6790/0 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b41c97e0 con 0x7f76b4141f20 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b413d3d0 con 0x7f76b413b3c0 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b4072120 con 0x7f76b4137360 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.149+0000 7f76baaae640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:57.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.150+0000 7f76baaae640 10 monclient(hunting): authenticate will time out at 1248.319517s 2024-03-29T10:41:57.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.153+0000 7f76b884a640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f76b413b3c0 0x7f76b4140d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:34689/0 (socket says 172.21.15.151:34689) 2024-03-29T10:41:57.136 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.153+0000 7f76b884a640 1 -- 172.21.15.151:0/901802210 learned_addr learned my addr 172.21.15.151:0/901802210 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 client.?.objecter ms_handle_connect 0x7f76b413b3c0 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264928625 0 0) 0x7f76b413d3d0 con 0x7f76b413b3c0 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:41:57.139 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.155+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a0004eb0 con 0x7f76b413b3c0 2024-03-29T10:41:57.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 client.?.objecter ms_handle_connect 0x7f76b4141f20 2024-03-29T10:41:57.140 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214141407 0 0) 0x7f76b41c97e0 con 0x7f76b4141f20 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b413d3d0 con 0x7f76b4141f20 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 client.?.objecter ms_handle_connect 0x7f76b4137360 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:57.141 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809959806 0 0) 0x7f76b4072120 con 0x7f76b4137360 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.156+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b41c97e0 con 0x7f76b4137360 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475453678 0 0) 0x7f76b41c97e0 con 0x7f76b4137360 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a0005620 con 0x7f76b4137360 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f76a40016d0 con 0x7f76b4137360 2024-03-29T10:41:57.142 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): dump: 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:57.147 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653624943 0 0) 0x7f76b413d3d0 con 0x7f76b4141f20 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a0005cd0 con 0x7f76b4141f20 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f76ac001a30 con 0x7f76b4141f20 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient(hunting): dump: 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.148 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 723976928 0 0) 0x7f76a0005620 con 0x7f76b4137360 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 >> v1:172.21.15.169:6789/0 conn(0x7f76b413b3c0 legacy=0x7f76b4140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 >> v1:172.21.15.151:6789/0 conn(0x7f76b4141f20 legacy=0x7f76b41c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 monclient: found mon.c 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40722b0 con 0x7f76b4137360 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:57.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 10 monclient: _renew_subs 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b413e380 con 0x7f76b4137360 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 10 monclient: _renew_subs 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b408bd80 con 0x7f76b4137360 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.157+0000 7f76baaae640 1 librados: init done 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.158+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76a4001bc0 con 0x7f76b4137360 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.158+0000 7f76b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.158+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f76a40016d0 con 0x7f76b4137360 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.158+0000 7f76b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:57.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.159+0000 7f76b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.159+0000 7f76b17fa640 10 monclient: dump: 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.160+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f76a4019500 con 0x7f76b4137360 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.161+0000 7f76b884a640 10 monclient: get_auth_request con 0x7f76a006e6f0 auth_method 0 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 10 client.4406.objecter ms_handle_connect 0x7f76a006e6f0 2024-03-29T10:41:57.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b2ffd640 4 set_mon_vals no callback set 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76baaae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f76a4052fb0 con 0x7f76b4137360 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 10 client.4406.objecter ms_dispatch 0x7f76b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.162+0000 7f76b17fa640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.163+0000 7f76baaae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.163+0000 7f76baaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.152 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.163+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694005080 con 0x7f76b4137360 2024-03-29T10:41:57.154 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.171+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76a404f250 con 0x7f76b4137360 2024-03-29T10:41:57.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.171+0000 7f76b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:57.155 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.171+0000 7f76b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:57.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.498+0000 7f76baaae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.498+0000 7f76baaae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.498+0000 7f76baaae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:41:57.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.498+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7694005330 con 0x7f76b4137360 2024-03-29T10:41:57.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.501+0000 7f76b17fa640 1 -- 172.21.15.151:0/901802210 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f76a404f250 con 0x7f76b4137360 2024-03-29T10:41:57.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.501+0000 7f76b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.501+0000 7f76b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:57.488 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:41:57.488 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:57.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.507+0000 7f76baaae640 10 librados: watch_flush enter 2024-03-29T10:41:57.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.507+0000 7f76baaae640 10 librados: watch_flush exit 2024-03-29T10:41:57.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.507+0000 7f76baaae640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T10:41:57.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.507+0000 7f76baaae640 10 client.4406.objecter successfully canceled tick 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.508+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 >> v1:172.21.15.151:6816/1549873000 conn(0x7f76a006e6f0 legacy=0x7f76a0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.508+0000 7f76baaae640 10 monclient: shutdown 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.508+0000 7f76baaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.508+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 >> v1:172.21.15.151:6790/0 conn(0x7f76b4137360 legacy=0x7f76b41eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.509+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 shutdown_connections 2024-03-29T10:41:57.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.509+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 >> 172.21.15.151:0/901802210 conn(0x7f76b40b35e0 msgr2=0x7f76b40a97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:57.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.511+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 shutdown_connections 2024-03-29T10:41:57.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.511+0000 7f76baaae640 1 -- 172.21.15.151:0/901802210 wait complete. 2024-03-29T10:41:57.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:41:57.511+0000 7f76baaae640 1 librados: shutdown 2024-03-29T10:41:57.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:04.323 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:04.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.490+0000 7f28365fa640 10 monclient: get_monmap_and_config 2024-03-29T10:42:04.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.490+0000 7f28365fa640 10 monclient: build_initial_monmap 2024-03-29T10:42:04.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.490+0000 7f28365fa640 10 monclient: monmap: 2024-03-29T10:42:04.478 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:04.491481+0000 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:04.491481+0000 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.491+0000 7f28365fa640 1 Processor -- start 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.491+0000 7f28365fa640 1 -- start start 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.491+0000 7f28365fa640 10 monclient: init 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:04.479 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2828029f00 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-b con 0x7f282802b090 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-c con 0x7f28280e2c40 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282802c0f0 con 0x7f2828029f00 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282802a430 con 0x7f28280e2c40 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282802a5c0 con 0x7f282802b090 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:04.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.492+0000 7f28365fa640 10 monclient(hunting): authenticate will time out at 1255.662209s 2024-03-29T10:42:04.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.496+0000 7f28355f8640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f2828029f00 0x7f28280e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:47457/0 (socket says 172.21.15.151:47457) 2024-03-29T10:42:04.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.496+0000 7f28355f8640 1 -- 172.21.15.151:0/3932996320 learned_addr learned my addr 172.21.15.151:0/3932996320 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707118270 0 0) 0x7f282802c0f0 con 0x7f2828029f00 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.499+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281c003520 con 0x7f2828029f00 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.500+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027254319 0 0) 0x7f281c003520 con 0x7f2828029f00 2024-03-29T10:42:04.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.500+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 >> v1:172.21.15.151:6790/0 conn(0x7f282802b090 legacy=0x7f28280e2530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:04.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 >> v1:172.21.15.169:6789/0 conn(0x7f28280e2c40 legacy=0x7f28280f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:04.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 monclient: found mon.noname-a 2024-03-29T10:42:04.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:04.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828006940 con 0x7f2828029f00 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f28200012f0 con 0x7f2828029f00 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: dump: 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.486 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28200014c0 con 0x7f2828029f00 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 1 -- 172.21.15.151:0/3932996320 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2820001910 con 0x7f2828029f00 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.502+0000 7f2827fff640 10 monclient: dump: 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:04.488 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.503+0000 7f28365fa640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.503+0000 7f28365fa640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.503+0000 7f28365fa640 4 set_mon_vals no callback set 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.504+0000 7f28365fa640 10 monclient: shutdown 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.504+0000 7f28365fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:04.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.504+0000 7f28365fa640 1 -- 172.21.15.151:0/3932996320 >> v1:172.21.15.151:6789/0 conn(0x7f2828029f00 legacy=0x7f28280e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.506+0000 7f28365fa640 1 -- 172.21.15.151:0/3932996320 shutdown_connections 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.506+0000 7f28365fa640 1 -- 172.21.15.151:0/3932996320 >> 172.21.15.151:0/3932996320 conn(0x7f28280058f0 msgr2=0x7f2828005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.506+0000 7f28365fa640 1 -- 172.21.15.151:0/3932996320 shutdown_connections 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.506+0000 7f28365fa640 1 -- 172.21.15.151:0/3932996320 wait complete. 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.507+0000 7f28365fa640 10 monclient: build_initial_monmap 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.507+0000 7f28365fa640 10 monclient: monmap: 2024-03-29T10:42:04.490 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:04.491 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.507+0000 7f28365fa640 1 librados: starting msgr at 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.507+0000 7f28365fa640 1 librados: starting objecter 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 1 Processor -- start 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 1 -- start start 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 1 librados: setting wanted keys 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 1 librados: calling monclient init 2024-03-29T10:42:04.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 10 monclient: init 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.508+0000 7f28365fa640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2828029f00 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-a con 0x7f282802b090 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient(hunting): picked mon.noname-c con 0x7f28280e2c40 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28280354c0 con 0x7f282802b090 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2828034590 con 0x7f2828029f00 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28280d48d0 con 0x7f28280e2c40 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:04.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.509+0000 7f28365fa640 10 monclient(hunting): authenticate will time out at 1255.678607s 2024-03-29T10:42:04.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.512+0000 7f28355f8640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f2828029f00 0x7f2828181ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58941/0 (socket says 172.21.15.151:58941) 2024-03-29T10:42:04.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.512+0000 7f28355f8640 1 -- 172.21.15.151:0/1203291266 learned_addr learned my addr 172.21.15.151:0/1203291266 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:04.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.512+0000 7f2825ffb640 10 client.?.objecter ms_handle_connect 0x7f2828029f00 2024-03-29T10:42:04.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.512+0000 7f2825ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:04.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.513+0000 7f2825ffb640 10 client.?.objecter ms_handle_connect 0x7f28280e2c40 2024-03-29T10:42:04.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.513+0000 7f2825ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.514+0000 7f2825ffb640 10 client.?.objecter ms_handle_connect 0x7f282802b090 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.514+0000 7f2825ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694949162 0 0) 0x7f2828034590 con 0x7f2828029f00 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2814004eb0 con 0x7f2828029f00 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740275009 0 0) 0x7f28280354c0 con 0x7f282802b090 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:42:04.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.515+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828034590 con 0x7f282802b090 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.516+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725044697 0 0) 0x7f28280d48d0 con 0x7f28280e2c40 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:42:04.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28280354c0 con 0x7f28280e2c40 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619204952 0 0) 0x7f2828034590 con 0x7f282802b090 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28280d48d0 con 0x7f282802b090 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f282c001830 con 0x7f282802b090 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:04.505 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665516899 0 0) 0x7f2814004eb0 con 0x7f2828029f00 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2828034590 con 0x7f2828029f00 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2820001890 con 0x7f2828029f00 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.520+0000 7f2825ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.506 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3986769009 0 0) 0x7f28280d48d0 con 0x7f282802b090 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 >> v1:172.21.15.151:6790/0 conn(0x7f28280e2c40 legacy=0x7f28280d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 >> v1:172.21.15.169:6789/0 conn(0x7f2828029f00 legacy=0x7f2828181ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 monclient: found mon.a 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280334d0 con 0x7f282802b090 2024-03-29T10:42:04.507 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:04.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:04.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:04.508 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f282c001d40 con 0x7f282802b090 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f282c003790 con 0x7f282802b090 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f2825ffb640 10 monclient: dump: 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:04.509 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f28277fe640 4 set_mon_vals no callback set 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f28365fa640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f28365fa640 10 monclient: _renew_subs 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f28365fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.521+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2828033720 con 0x7f282802b090 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.522+0000 7f28365fa640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.522+0000 7f28365fa640 10 monclient: _renew_subs 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.522+0000 7f28365fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.522+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28280d7cc0 con 0x7f282802b090 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.522+0000 7f28365fa640 1 librados: init done 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.523+0000 7f28365fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.510 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.523+0000 7f28365fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.523+0000 7f28365fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.523+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2828004cd0 con 0x7f282802b090 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.524+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f282c019760 con 0x7f282802b090 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f28355f8640 10 monclient: get_auth_request con 0x7f281406dc20 auth_method 0 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f282c0535a0 con 0x7f282802b090 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 10 client.4404.objecter ms_dispatch 0x7f2830000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T10:42:04.511 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.525+0000 7f2825ffb640 10 client.4404.objecter ms_handle_connect 0x7f281406dc20 2024-03-29T10:42:04.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.535+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f282c04f840 con 0x7f282802b090 2024-03-29T10:42:04.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.535+0000 7f2825ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.535+0000 7f2825ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:04.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.856+0000 7f28365fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.856+0000 7f28365fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.856+0000 7f28365fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:04.840 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.856+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28280086f0 con 0x7f282802b090 2024-03-29T10:42:04.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.858+0000 7f2825ffb640 1 -- 172.21.15.151:0/1203291266 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f282c04f840 con 0x7f282802b090 2024-03-29T10:42:04.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.858+0000 7f2825ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.858+0000 7f2825ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:04.842 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:04.842 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.864+0000 7f28365fa640 10 librados: watch_flush enter 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.864+0000 7f28365fa640 10 librados: watch_flush exit 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.864+0000 7f28365fa640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.864+0000 7f28365fa640 10 client.4404.objecter successfully canceled tick 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 >> v1:172.21.15.151:6816/1549873000 conn(0x7f281406dc20 legacy=0x7f28140700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 10 monclient: shutdown 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 >> v1:172.21.15.151:6789/0 conn(0x7f282802b090 legacy=0x7f28280d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 shutdown_connections 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 >> 172.21.15.151:0/1203291266 conn(0x7f28280058f0 msgr2=0x7f28280e46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:04.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 shutdown_connections 2024-03-29T10:42:04.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 -- 172.21.15.151:0/1203291266 wait complete. 2024-03-29T10:42:04.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:04.865+0000 7f28365fa640 1 librados: shutdown 2024-03-29T10:42:04.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:11.674 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:11.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.850+0000 7f636b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:42:11.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.850+0000 7f636b59e640 10 monclient: build_initial_monmap 2024-03-29T10:42:11.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.850+0000 7f636b59e640 10 monclient: monmap: 2024-03-29T10:42:11.834 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:11.851674+0000 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:11.851674+0000 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:11.835 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.852+0000 7f636b59e640 1 Processor -- start 2024-03-29T10:42:11.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.852+0000 7f636b59e640 1 -- start start 2024-03-29T10:42:11.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.852+0000 7f636b59e640 10 monclient: init 2024-03-29T10:42:11.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.854+0000 7f636b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:11.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.854+0000 7f636b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:11.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f6364006890 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:11.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f6364006c70 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:11.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f636402b4c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:11.841 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636402c830 con 0x7f636402b4c0 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636402afa0 con 0x7f6364006c70 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636402aba0 con 0x7f6364006890 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.855+0000 7f636b59e640 10 monclient(hunting): authenticate will time out at 1263.025043s 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.858+0000 7f636a59c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f6364006890 0x7f63640c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:43567/0 (socket says 172.21.15.151:43567) 2024-03-29T10:42:11.842 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.858+0000 7f636a59c640 1 -- 172.21.15.151:0/2081174660 learned_addr learned my addr 172.21.15.151:0/2081174660 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:11.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182519007 0 0) 0x7f636402c830 con 0x7f636402b4c0 2024-03-29T10:42:11.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6354003520 con 0x7f636402b4c0 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705992762 0 0) 0x7f636402aba0 con 0x7f6364006890 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636402c830 con 0x7f6364006890 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168404476 0 0) 0x7f636402afa0 con 0x7f6364006c70 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:42:11.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.862+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636402aba0 con 0x7f6364006c70 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327643138 0 0) 0x7f6354003520 con 0x7f636402b4c0 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 >> v1:172.21.15.151:6790/0 conn(0x7f6364006890 legacy=0x7f63640c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 >> v1:172.21.15.169:6789/0 conn(0x7f6364006c70 legacy=0x7f63640c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 monclient: found mon.noname-a 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636402ad30 con 0x7f636402b4c0 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 10 monclient: _finish_auth 0 2024-03-29T10:42:11.849 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f636c04f730 con 0x7f636402b4c0 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.863+0000 7f636b59e640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636959a640 10 monclient: dump: 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:11.850 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.864+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f636c075f10 con 0x7f636402b4c0 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636b59e640 4 set_mon_vals no callback set 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636959a640 1 -- 172.21.15.151:0/2081174660 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f636c0758b0 con 0x7f636402b4c0 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.865+0000 7f636959a640 10 monclient: dump: 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.851 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.866+0000 7f636b59e640 10 monclient: shutdown 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.866+0000 7f636b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:11.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.866+0000 7f636b59e640 1 -- 172.21.15.151:0/2081174660 >> v1:172.21.15.151:6789/0 conn(0x7f636402b4c0 legacy=0x7f63640cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.866+0000 7f636b59e640 1 -- 172.21.15.151:0/2081174660 shutdown_connections 2024-03-29T10:42:11.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.866+0000 7f636b59e640 1 -- 172.21.15.151:0/2081174660 >> 172.21.15.151:0/2081174660 conn(0x7f63640058f0 msgr2=0x7f6364005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:11.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.867+0000 7f636b59e640 1 -- 172.21.15.151:0/2081174660 shutdown_connections 2024-03-29T10:42:11.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.867+0000 7f636b59e640 1 -- 172.21.15.151:0/2081174660 wait complete. 2024-03-29T10:42:11.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.868+0000 7f636b59e640 10 monclient: build_initial_monmap 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.868+0000 7f636b59e640 10 monclient: monmap: 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:11.859 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 librados: starting msgr at 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 librados: starting objecter 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 Processor -- start 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 -- start start 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 librados: setting wanted keys 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 1 librados: calling monclient init 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.869+0000 7f636b59e640 10 monclient: init 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.870+0000 7f636b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.870+0000 7f636b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f6364006890 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f636402b4c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f63640b4c40 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:11.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636402cf40 con 0x7f63640b4c40 2024-03-29T10:42:11.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640c2690 con 0x7f6364006890 2024-03-29T10:42:11.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640b4010 con 0x7f636402b4c0 2024-03-29T10:42:11.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:11.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.871+0000 7f636b59e640 10 monclient(hunting): authenticate will time out at 1263.040869s 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.877+0000 7f636ad9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f63640b4c40 0x7f63640afba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:47286/0 (socket says 172.21.15.151:47286) 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.877+0000 7f636ad9d640 1 -- 172.21.15.151:0/2152268435 learned_addr learned my addr 172.21.15.151:0/2152268435 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.877+0000 7f635affd640 10 client.?.objecter ms_handle_connect 0x7f636402b4c0 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.877+0000 7f635affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 client.?.objecter ms_handle_connect 0x7f6364006890 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855412776 0 0) 0x7f63640c2690 con 0x7f6364006890 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f6364006890 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789639590 0 0) 0x7f63640b4010 con 0x7f636402b4c0 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63640c2690 con 0x7f636402b4c0 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 client.?.objecter ms_handle_connect 0x7f63640b4c40 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790651733 0 0) 0x7f634c004eb0 con 0x7f6364006890 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63640b4010 con 0x7f6364006890 2024-03-29T10:42:11.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f635c001810 con 0x7f6364006890 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.878+0000 7f635affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): dump: 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658609251 0 0) 0x7f63640c2690 con 0x7f636402b4c0 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f634c004eb0 con 0x7f636402b4c0 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f63600017e0 con 0x7f636402b4c0 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient(hunting): dump: 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:11.869 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1863079375 0 0) 0x7f634c004eb0 con 0x7f636402b4c0 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 >> v1:172.21.15.169:6789/0 conn(0x7f6364006890 legacy=0x7f63640b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 >> v1:172.21.15.151:6789/0 conn(0x7f63640b4c40 legacy=0x7f63640afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 monclient: found mon.c 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640be200 con 0x7f636402b4c0 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.879+0000 7f635affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.880+0000 7f636b59e640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:42:11.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.880+0000 7f636b59e640 10 monclient: _renew_subs 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.880+0000 7f636b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.880+0000 7f636b59e640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63640c3810 con 0x7f636402b4c0 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.880+0000 7f636b59e640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6360001c70 con 0x7f636402b4c0 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f63600037b0 con 0x7f636402b4c0 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f635affd640 10 monclient: dump: 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.871 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f6368d99640 4 set_mon_vals no callback set 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f636b59e640 10 monclient: _renew_subs 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f636b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f636b59e640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63640b12c0 con 0x7f636402b4c0 2024-03-29T10:42:11.872 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.881+0000 7f636b59e640 1 librados: init done 2024-03-29T10:42:11.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.883+0000 7f6358ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.883+0000 7f6358ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.883+0000 7f6358ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.883+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344005080 con 0x7f636402b4c0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.885+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f6360019780 con 0x7f636402b4c0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f6360083600 con 0x7f636402b4c0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f635affd640 10 client.4415.objecter ms_dispatch 0x7f636c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f635affd640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f635affd640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f635affd640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.886+0000 7f636a59c640 10 monclient: get_auth_request con 0x7f634c06df80 auth_method 0 2024-03-29T10:42:11.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.888+0000 7f635affd640 10 client.4415.objecter ms_handle_connect 0x7f634c06df80 2024-03-29T10:42:11.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.892+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f636004ed10 con 0x7f636402b4c0 2024-03-29T10:42:11.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.892+0000 7f635affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:11.892+0000 7f635affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:12.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.231+0000 7f6358ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:12.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.231+0000 7f6358ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:12.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.231+0000 7f6358ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:12.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.231+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6344005330 con 0x7f636402b4c0 2024-03-29T10:42:12.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.235+0000 7f635affd640 1 -- 172.21.15.151:0/2152268435 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f6360050e10 con 0x7f636402b4c0 2024-03-29T10:42:12.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.235+0000 7f635affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:12.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.235+0000 7f635affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:12.219 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:12.220 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.239+0000 7f6358ff9640 10 librados: watch_flush enter 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.239+0000 7f6358ff9640 10 librados: watch_flush exit 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.239+0000 7f6358ff9640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.239+0000 7f6358ff9640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 >> v1:172.21.15.151:6816/1549873000 conn(0x7f634c06df80 legacy=0x7f634c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 10 monclient: shutdown 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 >> v1:172.21.15.151:6790/0 conn(0x7f636402b4c0 legacy=0x7f63640c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 shutdown_connections 2024-03-29T10:42:12.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.243+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 >> 172.21.15.151:0/2152268435 conn(0x7f63640058f0 msgr2=0x7f63640c0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:12.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.244+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 shutdown_connections 2024-03-29T10:42:12.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.244+0000 7f6358ff9640 1 -- 172.21.15.151:0/2152268435 wait complete. 2024-03-29T10:42:12.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:12.245+0000 7f6358ff9640 1 librados: shutdown 2024-03-29T10:42:12.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:19.055 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:19.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.219+0000 7f5389c4a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.219+0000 7f5389c4a640 10 monclient: build_initial_monmap 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.219+0000 7f5389c4a640 10 monclient: monmap: 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:19.220266+0000 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:19.220266+0000 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:19.203 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.220+0000 7f5389c4a640 1 Processor -- start 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.220+0000 7f5389c4a640 1 -- start start 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.220+0000 7f5389c4a640 10 monclient: init 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53841457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:19.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f538413f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f53840a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840aaef0 con 0x7f538413f650 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384145bd0 con 0x7f53841457f0 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384145eb0 con 0x7f53840a9d70 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:19.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.221+0000 7f5389c4a640 10 monclient(hunting): authenticate will time out at 1270.391182s 2024-03-29T10:42:19.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.223+0000 7f5383fff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f53840a9d70 0x7f538414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:33554/0 (socket says 172.21.15.151:33554) 2024-03-29T10:42:19.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.223+0000 7f5383fff640 1 -- 172.21.15.151:0/2991979833 learned_addr learned my addr 172.21.15.151:0/2991979833 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491052185 0 0) 0x7f5384145eb0 con 0x7f53840a9d70 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.224+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c003520 con 0x7f53840a9d70 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.225+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416999664 0 0) 0x7f536c003520 con 0x7f53840a9d70 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.225+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 >> v1:172.21.15.169:6789/0 conn(0x7f53841457f0 legacy=0x7f538413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:19.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.225+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 >> v1:172.21.15.151:6789/0 conn(0x7f538413f650 legacy=0x7f53840b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 1 monclient: found mon.noname-b 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384146340 con 0x7f53840a9d70 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f537c001a10 con 0x7f53840a9d70 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.226+0000 7f53827fc640 10 monclient: dump: 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:19.212 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f537c001c70 con 0x7f53840a9d70 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 1 -- 172.21.15.151:0/2991979833 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f537c0020c0 con 0x7f53840a9d70 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f53827fc640 10 monclient: dump: 2024-03-29T10:42:19.213 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f5389c4a640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f5389c4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.227+0000 7f5389c4a640 4 set_mon_vals no callback set 2024-03-29T10:42:19.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.228+0000 7f5389c4a640 10 monclient: shutdown 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.228+0000 7f5389c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.228+0000 7f5389c4a640 1 -- 172.21.15.151:0/2991979833 >> v1:172.21.15.151:6790/0 conn(0x7f53840a9d70 legacy=0x7f538414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.228+0000 7f5389c4a640 1 -- 172.21.15.151:0/2991979833 shutdown_connections 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.228+0000 7f5389c4a640 1 -- 172.21.15.151:0/2991979833 >> 172.21.15.151:0/2991979833 conn(0x7f5384066a00 msgr2=0x7f5384066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.231+0000 7f5389c4a640 1 -- 172.21.15.151:0/2991979833 shutdown_connections 2024-03-29T10:42:19.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.231+0000 7f5389c4a640 1 -- 172.21.15.151:0/2991979833 wait complete. 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.232+0000 7f5389c4a640 10 monclient: build_initial_monmap 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.232+0000 7f5389c4a640 10 monclient: monmap: 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:19.216 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.233+0000 7f5389c4a640 1 librados: starting msgr at 2024-03-29T10:42:19.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.233+0000 7f5389c4a640 1 librados: starting objecter 2024-03-29T10:42:19.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.233+0000 7f5389c4a640 1 Processor -- start 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.233+0000 7f5389c4a640 1 -- start start 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.233+0000 7f5389c4a640 1 librados: setting wanted keys 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.234+0000 7f5389c4a640 1 librados: calling monclient init 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.234+0000 7f5389c4a640 10 monclient: init 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.234+0000 7f5389c4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:19.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.235+0000 7f5389c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:19.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.235+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f53840a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:19.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f538413f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f53841457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384141290 con 0x7f53840a9d70 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538413dee0 con 0x7f538413f650 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384141500 con 0x7f53841457f0 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f5389c4a640 10 monclient(hunting): authenticate will time out at 1270.408000s 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f53837fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f53840a9d70 0x7f538413c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43495/0 (socket says 172.21.15.151:43495) 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.238+0000 7f53837fe640 1 -- 172.21.15.151:0/3109965943 learned_addr learned my addr 172.21.15.151:0/3109965943 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 client.?.objecter ms_handle_connect 0x7f53840a9d70 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 client.?.objecter ms_handle_connect 0x7f53841457f0 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871994901 0 0) 0x7f5384141290 con 0x7f53840a9d70 2024-03-29T10:42:19.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.239+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370004eb0 con 0x7f53840a9d70 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136461667 0 0) 0x7f5370004eb0 con 0x7f53840a9d70 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5384141290 con 0x7f53840a9d70 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5374003fe0 con 0x7f53840a9d70 2024-03-29T10:42:19.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): dump: 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193229223 0 0) 0x7f5384141500 con 0x7f53841457f0 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370004eb0 con 0x7f53841457f0 2024-03-29T10:42:19.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.240+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2174632378 0 0) 0x7f5370004eb0 con 0x7f53841457f0 2024-03-29T10:42:19.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.241+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5384141500 con 0x7f53841457f0 2024-03-29T10:42:19.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.241+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f537c001830 con 0x7f53841457f0 2024-03-29T10:42:19.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.241+0000 7f536bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.242+0000 7f536bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.242+0000 7f536bfff640 10 monclient(hunting): dump: 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:19.229 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:19.230 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.242+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 373732631 0 0) 0x7f5384141290 con 0x7f53840a9d70 2024-03-29T10:42:19.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.242+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 >> v1:172.21.15.151:6790/0 conn(0x7f53841457f0 legacy=0x7f5384132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.242+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 >> v1:172.21.15.169:6789/0 conn(0x7f538413f650 legacy=0x7f538413fe30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:19.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:19.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 1 monclient: found mon.a 2024-03-29T10:42:19.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538413db90 con 0x7f53840a9d70 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 10 monclient: _finish_auth 0 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f536bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 10 monclient: _renew_subs 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5384142cb0 con 0x7f53840a9d70 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 10 monclient: _renew_subs 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53841e2e00 con 0x7f53840a9d70 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.243+0000 7f5389c4a640 1 librados: init done 2024-03-29T10:42:19.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.246+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5374000ed0 con 0x7f53840a9d70 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.246+0000 7f536bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.246+0000 7f5381ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.247+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f5374004140 con 0x7f53840a9d70 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.247+0000 7f536bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.247+0000 7f536bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.247+0000 7f536bfff640 10 monclient: dump: 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:19.233 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:19.234 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:19.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.247+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f5374001360 con 0x7f53840a9d70 2024-03-29T10:42:19.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.248+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f5374084b70 con 0x7f53840a9d70 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.248+0000 7f5389c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.248+0000 7f5389c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.248+0000 7f5389c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.248+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5364005080 con 0x7f53840a9d70 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.250+0000 7f536bfff640 10 client.4413.objecter ms_dispatch 0x7f5384000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.250+0000 7f536bfff640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.250+0000 7f536bfff640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.250+0000 7f536bfff640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.251+0000 7f5382ffd640 10 monclient: get_auth_request con 0x7f537006e070 auth_method 0 2024-03-29T10:42:19.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.251+0000 7f536bfff640 10 client.4413.objecter ms_handle_connect 0x7f537006e070 2024-03-29T10:42:19.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.259+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53740504d0 con 0x7f53840a9d70 2024-03-29T10:42:19.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.259+0000 7f536bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:19.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.259+0000 7f536bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:19.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.600+0000 7f5389c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.600+0000 7f5389c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.600+0000 7f5389c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:19.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.600+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5364005330 con 0x7f53840a9d70 2024-03-29T10:42:19.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.602+0000 7f536bfff640 1 -- 172.21.15.151:0/3109965943 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f5374052400 con 0x7f53840a9d70 2024-03-29T10:42:19.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.602+0000 7f536bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.602+0000 7f536bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:19.586 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:19.587 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.606+0000 7f5389c4a640 10 librados: watch_flush enter 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.606+0000 7f5389c4a640 10 librados: watch_flush exit 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.606+0000 7f5389c4a640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.606+0000 7f5389c4a640 10 client.4413.objecter successfully canceled tick 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 >> v1:172.21.15.151:6816/1549873000 conn(0x7f537006e070 legacy=0x7f5370070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.590 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 10 monclient: shutdown 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 >> v1:172.21.15.151:6789/0 conn(0x7f53840a9d70 legacy=0x7f538413c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 shutdown_connections 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 >> 172.21.15.151:0/3109965943 conn(0x7f5384066a00 msgr2=0x7f53840b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 shutdown_connections 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 -- 172.21.15.151:0/3109965943 wait complete. 2024-03-29T10:42:19.591 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:19.607+0000 7f5389c4a640 1 librados: shutdown 2024-03-29T10:42:19.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:26.418 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:26.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.562+0000 7f4c1ac27640 10 monclient: get_monmap_and_config 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.562+0000 7f4c1ac27640 10 monclient: build_initial_monmap 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.562+0000 7f4c1ac27640 10 monclient: monmap: 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:26.564535+0000 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:26.564535+0000 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:26.547 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.563+0000 7f4c1ac27640 1 Processor -- start 2024-03-29T10:42:26.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.563+0000 7f4c1ac27640 1 -- start start 2024-03-29T10:42:26.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.563+0000 7f4c1ac27640 10 monclient: init 2024-03-29T10:42:26.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-c con 0x7f4c14074650 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-a con 0x7f4c14072e80 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1406f510 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1406f080 con 0x7f4c14072e80 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c14070900 con 0x7f4c14074650 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c14143d10 con 0x7f4c1406f510 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c1ac27640 10 monclient(hunting): authenticate will time out at 1277.736071s 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c189c3640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4c14074650 0x7f4c14143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:33152/0 (socket says 172.21.15.151:33152) 2024-03-29T10:42:26.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.565+0000 7f4c189c3640 1 -- 172.21.15.151:0/2785595041 learned_addr learned my addr 172.21.15.151:0/2785595041 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989929072 0 0) 0x7f4c1406f080 con 0x7f4c14072e80 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04003520 con 0x7f4c14072e80 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658135359 0 0) 0x7f4c14070900 con 0x7f4c14074650 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:42:26.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.566+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1406f080 con 0x7f4c14074650 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1124954569 0 0) 0x7f4c1406f080 con 0x7f4c14074650 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 >> v1:172.21.15.151:6790/0 conn(0x7f4c1406f510 legacy=0x7f4c14151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 >> v1:172.21.15.151:6789/0 conn(0x7f4c14072e80 legacy=0x7f4c141499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 monclient: found mon.noname-c 2024-03-29T10:42:26.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1406e8d0 con 0x7f4c14074650 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c080015f0 con 0x7f4c14074650 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.567+0000 7f4c1ac27640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c1ac27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: dump: 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.555 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c1ac27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c08001750 con 0x7f4c14074650 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 1 -- 172.21.15.151:0/2785595041 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c08001b60 con 0x7f4c14074650 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:26.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c137fe640 10 monclient: dump: 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c1ac27640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.568+0000 7f4c1ac27640 4 set_mon_vals no callback set 2024-03-29T10:42:26.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.569+0000 7f4c1ac27640 10 monclient: shutdown 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.569+0000 7f4c1ac27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.569+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2785595041 >> v1:172.21.15.169:6789/0 conn(0x7f4c14074650 legacy=0x7f4c14143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.569+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2785595041 shutdown_connections 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.569+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2785595041 >> 172.21.15.151:0/2785595041 conn(0x7f4c140b4200 msgr2=0x7f4c140b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.570+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2785595041 shutdown_connections 2024-03-29T10:42:26.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.570+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2785595041 wait complete. 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.572+0000 7f4c1ac27640 10 monclient: build_initial_monmap 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.572+0000 7f4c1ac27640 10 monclient: monmap: 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:26.564 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 librados: starting msgr at 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 librados: starting objecter 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 Processor -- start 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 -- start start 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 librados: setting wanted keys 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 1 librados: calling monclient init 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.573+0000 7f4c1ac27640 10 monclient: init 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c1ac27640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c1ac27640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-a con 0x7f4c1406f510 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-c con 0x7f4c14074650 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c1ac27640 10 monclient(hunting): picked mon.noname-b con 0x7f4c141d1680 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:26.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.574+0000 7f4c189c3640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4c1406f510 0x7f4c1413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:34420/0 (socket says 172.21.15.151:34420) 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c189c3640 1 -- 172.21.15.151:0/2036847762 learned_addr learned my addr 172.21.15.151:0/2036847762 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c1ac27640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c141d2130 con 0x7f4c1406f510 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c141d1c10 con 0x7f4c141d1680 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c14140120 con 0x7f4c14074650 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c1ac27640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c1ac27640 10 monclient(hunting): authenticate will time out at 1277.745684s 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter ms_handle_connect 0x7f4c1406f510 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter ms_handle_connect 0x7f4c141d1680 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter ms_handle_connect 0x7f4c14074650 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071742952 0 0) 0x7f4c141d1c10 con 0x7f4c141d1680 2024-03-29T10:42:26.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.575+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8004eb0 con 0x7f4c141d1680 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345798259 0 0) 0x7f4c14140120 con 0x7f4c14074650 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c141d1c10 con 0x7f4c14074650 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534869437 0 0) 0x7f4c141d2130 con 0x7f4c1406f510 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14140120 con 0x7f4c1406f510 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3243985514 0 0) 0x7f4bf8004eb0 con 0x7f4c141d1680 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c141d2130 con 0x7f4c141d1680 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c0c001b70 con 0x7f4c141d1680 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): dump: 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:26.568 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2911036142 0 0) 0x7f4c141d1c10 con 0x7f4c14074650 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bf8004eb0 con 0x7f4c14074650 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2830651031 0 0) 0x7f4c141d2130 con 0x7f4c141d1680 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 >> v1:172.21.15.151:6790/0 conn(0x7f4c14074650 legacy=0x7f4c141cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 >> v1:172.21.15.151:6789/0 conn(0x7f4c1406f510 legacy=0x7f4c1413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 monclient: found mon.b 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c141402b0 con 0x7f4c141d1680 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c1ac27640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c1ac27640 10 monclient: _renew_subs 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c1ac27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.576+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c14146b40 con 0x7f4c141d1680 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c1ac27640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c1ac27640 10 monclient: _renew_subs 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c1ac27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c141cf890 con 0x7f4c141d1680 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c1ac27640 1 librados: init done 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c0c001cd0 con 0x7f4c141d1680 2024-03-29T10:42:26.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c0c003b10 con 0x7f4c141d1680 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c117fa640 10 monclient: dump: 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:26.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.577+0000 7f4c12ffd640 4 set_mon_vals no callback set 2024-03-29T10:42:26.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.578+0000 7f4c1ac27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.579+0000 7f4c1ac27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.579+0000 7f4c1ac27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.579+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf4005080 con 0x7f4c141d1680 2024-03-29T10:42:26.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.579+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4c0c019ae0 con 0x7f4c141d1680 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.580+0000 7f4c189c3640 10 monclient: get_auth_request con 0x7f4bf806df30 auth_method 0 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.580+0000 7f4c117fa640 10 client.4429.objecter ms_handle_connect 0x7f4bf806df30 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.581+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4c0c083aa0 con 0x7f4c141d1680 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.581+0000 7f4c117fa640 10 client.4429.objecter ms_dispatch 0x7f4c14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.581+0000 7f4c117fa640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.581+0000 7f4c117fa640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:26.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.581+0000 7f4c117fa640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T10:42:26.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.590+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c0c04f1b0 con 0x7f4c141d1680 2024-03-29T10:42:26.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.590+0000 7f4c117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.590+0000 7f4c117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:26.955 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.971+0000 7f4c1ac27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.971+0000 7f4c1ac27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.971+0000 7f4c1ac27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.971+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bf4005330 con 0x7f4c141d1680 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.972+0000 7f4c117fa640 1 -- 172.21.15.151:0/2036847762 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4c0c0512b0 con 0x7f4c141d1680 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.972+0000 7f4c117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.972+0000 7f4c117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:26.957 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:26.957 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 10 librados: watch_flush enter 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 10 librados: watch_flush exit 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 10 client.4429.objecter successfully canceled tick 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4bf806df30 legacy=0x7f4bf80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 10 monclient: shutdown 2024-03-29T10:42:26.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:26.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 >> v1:172.21.15.169:6789/0 conn(0x7f4c141d1680 legacy=0x7f4c141e6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 shutdown_connections 2024-03-29T10:42:26.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 >> 172.21.15.151:0/2036847762 conn(0x7f4c140b4200 msgr2=0x7f4c140b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:26.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.976+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 shutdown_connections 2024-03-29T10:42:26.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.977+0000 7f4c1ac27640 1 -- 172.21.15.151:0/2036847762 wait complete. 2024-03-29T10:42:26.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:26.977+0000 7f4c1ac27640 1 librados: shutdown 2024-03-29T10:42:26.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:33.781 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.931+0000 7f776e97b640 10 monclient: get_monmap_and_config 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.931+0000 7f776e97b640 10 monclient: build_initial_monmap 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.931+0000 7f776e97b640 10 monclient: monmap: 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.915 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:33.933045+0000 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:33.933045+0000 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:33.916 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.932+0000 7f776e97b640 1 Processor -- start 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.932+0000 7f776e97b640 1 -- start start 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.932+0000 7f776e97b640 10 monclient: init 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-a con 0x7f77681457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-c con 0x7f776813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-b con 0x7f77680a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776cf18640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f77680a9d70 0x7f776814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60094/0 (socket says 172.21.15.151:60094) 2024-03-29T10:42:33.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776cf18640 1 -- 172.21.15.151:0/3241138957 learned_addr learned my addr 172.21.15.151:0/3241138957 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f77677fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f776813f650 0x7f77680b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58350/0 (socket says 172.21.15.151:58350) 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680aaef0 con 0x7f77681457f0 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768145bd0 con 0x7f776813f650 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768145eb0 con 0x7f77680a9d70 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f776e97b640 10 monclient(hunting): authenticate will time out at 1285.104076s 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394920241 0 0) 0x7f7768145bd0 con 0x7f776813f650 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7750003520 con 0x7f776813f650 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995976498 0 0) 0x7f7768145eb0 con 0x7f77680a9d70 2024-03-29T10:42:33.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.933+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768145bd0 con 0x7f77680a9d70 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.934+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415907199 0 0) 0x7f7768145bd0 con 0x7f77680a9d70 2024-03-29T10:42:33.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.934+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 >> v1:172.21.15.169:6789/0 conn(0x7f776813f650 legacy=0x7f77680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.934+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 >> v1:172.21.15.151:6789/0 conn(0x7f77681457f0 legacy=0x7f776813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 1 monclient: found mon.noname-b 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768146340 con 0x7f77680a9d70 2024-03-29T10:42:33.922 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f77600019f0 con 0x7f77680a9d70 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: dump: 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:33.923 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f776e97b640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f776e97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7760001bd0 con 0x7f77680a9d70 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 1 -- 172.21.15.151:0/3241138957 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7760001fe0 con 0x7f77680a9d70 2024-03-29T10:42:33.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f7766ffd640 10 monclient: dump: 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f776e97b640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:33.925 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.937+0000 7f776e97b640 4 set_mon_vals no callback set 2024-03-29T10:42:33.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.938+0000 7f776e97b640 10 monclient: shutdown 2024-03-29T10:42:33.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.938+0000 7f776e97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:33.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.938+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 >> v1:172.21.15.151:6790/0 conn(0x7f77680a9d70 legacy=0x7f776814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.938+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 shutdown_connections 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.938+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 >> 172.21.15.151:0/3241138957 conn(0x7f7768066a00 msgr2=0x7f7768066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.939+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 shutdown_connections 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.939+0000 7f776e97b640 1 -- 172.21.15.151:0/3241138957 wait complete. 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.940+0000 7f776e97b640 10 monclient: build_initial_monmap 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.940+0000 7f776e97b640 10 monclient: monmap: 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:33.935 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 librados: starting msgr at 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 librados: starting objecter 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 Processor -- start 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 -- start start 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 librados: setting wanted keys 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 1 librados: calling monclient init 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.941+0000 7f776e97b640 10 monclient: init 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.942+0000 7f776e97b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.942+0000 7f776e97b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.942+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-a con 0x7f77680a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-c con 0x7f776813f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f77677fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f776813f650 0x7f77681d43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60974/0 (socket says 172.21.15.151:60974) 2024-03-29T10:42:33.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f77677fe640 1 -- 172.21.15.151:0/653660596 learned_addr learned my addr 172.21.15.151:0/653660596 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f7764ff9640 10 client.?.objecter ms_handle_connect 0x7f776813f650 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f7764ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f776e97b640 10 monclient(hunting): picked mon.noname-b con 0x7f77681457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768141c80 con 0x7f77680a9d70 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f7764ff9640 10 client.?.objecter ms_handle_connect 0x7f77681457f0 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f7764ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77681d7720 con 0x7f77681457f0 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.943+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768141ef0 con 0x7f776813f650 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f776e97b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f776e97b640 10 monclient(hunting): authenticate will time out at 1285.114587s 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789303549 0 0) 0x7f7768141ef0 con 0x7f776813f650 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754004eb0 con 0x7f776813f650 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403075368 0 0) 0x7f77681d7720 con 0x7f77681457f0 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768141ef0 con 0x7f77681457f0 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014812666 0 0) 0x7f7754004eb0 con 0x7f776813f650 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77681d7720 con 0x7f776813f650 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f775c001950 con 0x7f776813f650 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286333573 0 0) 0x7f7768141ef0 con 0x7f77681457f0 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7754004eb0 con 0x7f77681457f0 2024-03-29T10:42:33.939 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7760001ee0 con 0x7f77681457f0 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.944+0000 7f7764ff9640 10 monclient(hunting): dump: 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.940 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377164539 0 0) 0x7f77681d7720 con 0x7f776813f650 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 >> v1:172.21.15.169:6789/0 conn(0x7f77681457f0 legacy=0x7f7768142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 >> v1:172.21.15.151:6789/0 conn(0x7f77680a9d70 legacy=0x7f77681d7010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 1 monclient: found mon.c 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768140230 con 0x7f776813f650 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 10 monclient: _finish_auth 0 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f7764ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 10 monclient: _renew_subs 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77681d4ae0 con 0x7f776813f650 2024-03-29T10:42:33.941 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 10 monclient: _renew_subs 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.945+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776813ff80 con 0x7f776813f650 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.946+0000 7f776e97b640 1 librados: init done 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f776e97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f776e97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f776e97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7768140470 con 0x7f776813f650 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f775c0047e0 con 0x7f776813f650 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f775c004af0 con 0x7f776813f650 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.942 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f7764ff9640 10 monclient: dump: 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.947+0000 7f77667fc640 4 set_mon_vals no callback set 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.949+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f775c01ab40 con 0x7f776813f650 2024-03-29T10:42:33.943 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7767fff640 10 monclient: get_auth_request con 0x7f775406de30 auth_method 0 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f775c0848a0 con 0x7f776813f650 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 10 client.4433.objecter ms_dispatch 0x7f7768000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.950+0000 7f7764ff9640 10 client.4433.objecter ms_handle_connect 0x7f775406de30 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.957+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f775c0500f0 con 0x7f776813f650 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.957+0000 7f7764ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.944 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:33.957+0000 7f7764ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:34.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.256+0000 7f776e97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.256+0000 7f776e97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.256+0000 7f776e97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:34.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.256+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7768140470 con 0x7f776813f650 2024-03-29T10:42:34.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.257+0000 7f7764ff9640 1 -- 172.21.15.151:0/653660596 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f7768140470 con 0x7f776813f650 2024-03-29T10:42:34.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.257+0000 7f7764ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.242 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.257+0000 7f7764ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:34.242 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:34.242 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 10 librados: watch_flush enter 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 10 librados: watch_flush exit 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 10 client.4433.objecter successfully canceled tick 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 >> v1:172.21.15.151:6816/1549873000 conn(0x7f775406de30 legacy=0x7f77540702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:34.245 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 10 monclient: shutdown 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 >> v1:172.21.15.151:6790/0 conn(0x7f776813f650 legacy=0x7f77681d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 shutdown_connections 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.260+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 >> 172.21.15.151:0/653660596 conn(0x7f7768066a00 msgr2=0x7f77680b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.261+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 shutdown_connections 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.261+0000 7f776e97b640 1 -- 172.21.15.151:0/653660596 wait complete. 2024-03-29T10:42:34.246 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:34.261+0000 7f776e97b640 1 librados: shutdown 2024-03-29T10:42:34.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:41.066 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:41.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.228+0000 7f3bf57cf640 10 monclient: get_monmap_and_config 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.228+0000 7f3bf57cf640 10 monclient: build_initial_monmap 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.228+0000 7f3bf57cf640 10 monclient: monmap: 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:41.229350+0000 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:41.229350+0000 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:41.212 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.229+0000 7f3bf57cf640 1 Processor -- start 2024-03-29T10:42:41.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.230+0000 7f3bf57cf640 1 -- start start 2024-03-29T10:42:41.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.230+0000 7f3bf57cf640 10 monclient: init 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf01457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf013f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf00a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf00aaef0 con 0x7f3bf00a9d70 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf0145bd0 con 0x7f3bf01457f0 2024-03-29T10:42:41.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf0145eb0 con 0x7f3bf013f650 2024-03-29T10:42:41.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:41.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.231+0000 7f3bf57cf640 10 monclient(hunting): authenticate will time out at 1292.400859s 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bee7fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f3bf013f650 0x7f3bf00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:37480/0 (socket says 172.21.15.151:37480) 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bee7fc640 1 -- 172.21.15.151:0/130906936 learned_addr learned my addr 172.21.15.151:0/130906936 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963150965 0 0) 0x7f3bf0145eb0 con 0x7f3bf013f650 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bdc003520 con 0x7f3bf013f650 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496744472 0 0) 0x7f3bdc003520 con 0x7f3bf013f650 2024-03-29T10:42:41.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.232+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 >> v1:172.21.15.169:6789/0 conn(0x7f3bf01457f0 legacy=0x7f3bf013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:41.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 >> v1:172.21.15.151:6789/0 conn(0x7f3bf00a9d70 legacy=0x7f3bf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:41.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:41.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 monclient: found mon.noname-b 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0146340 con 0x7f3bf013f650 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3be8001670 con 0x7f3bf013f650 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: dump: 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.221 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3be80018d0 con 0x7f3bf013f650 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 1 -- 172.21.15.151:0/130906936 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3be8001d20 con 0x7f3bf013f650 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bedffb640 10 monclient: dump: 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:41.222 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bf57cf640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bf57cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.235+0000 7f3bf57cf640 4 set_mon_vals no callback set 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.236+0000 7f3bf57cf640 10 monclient: shutdown 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.236+0000 7f3bf57cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.236+0000 7f3bf57cf640 1 -- 172.21.15.151:0/130906936 >> v1:172.21.15.151:6790/0 conn(0x7f3bf013f650 legacy=0x7f3bf00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.236+0000 7f3bf57cf640 1 -- 172.21.15.151:0/130906936 shutdown_connections 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.236+0000 7f3bf57cf640 1 -- 172.21.15.151:0/130906936 >> 172.21.15.151:0/130906936 conn(0x7f3bf0066a00 msgr2=0x7f3bf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.237+0000 7f3bf57cf640 1 -- 172.21.15.151:0/130906936 shutdown_connections 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.237+0000 7f3bf57cf640 1 -- 172.21.15.151:0/130906936 wait complete. 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.238+0000 7f3bf57cf640 10 monclient: build_initial_monmap 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.238+0000 7f3bf57cf640 10 monclient: monmap: 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:41.227 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.238+0000 7f3bf57cf640 1 librados: starting msgr at 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.238+0000 7f3bf57cf640 1 librados: starting objecter 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.238+0000 7f3bf57cf640 1 Processor -- start 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.239+0000 7f3bf57cf640 1 -- start start 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.239+0000 7f3bf57cf640 1 librados: setting wanted keys 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.239+0000 7f3bf57cf640 1 librados: calling monclient init 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.239+0000 7f3bf57cf640 10 monclient: init 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf00a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:41.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf01457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf01363b0 con 0x7f3bf00a9d70 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf0136cb0 con 0x7f3bf013f650 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf0136f90 con 0x7f3bf01457f0 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bf57cf640 10 monclient(hunting): authenticate will time out at 1292.410155s 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bef7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f3bf01457f0 0x7f3bf01e64a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:38758/0 (socket says 172.21.15.151:38758) 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.240+0000 7f3bef7fe640 1 -- 172.21.15.151:0/43406365 learned_addr learned my addr 172.21.15.151:0/43406365 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 client.?.objecter ms_handle_connect 0x7f3bf01457f0 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736299669 0 0) 0x7f3bf0136f90 con 0x7f3bf01457f0 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:41.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.241+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0004eb0 con 0x7f3bf01457f0 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677905173 0 0) 0x7f3bd0004eb0 con 0x7f3bf01457f0 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bf0136f90 con 0x7f3bf01457f0 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3be0001c60 con 0x7f3bf01457f0 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 10 monclient(hunting): dump: 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.230 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3417402886 0 0) 0x7f3bf0136f90 con 0x7f3bf01457f0 2024-03-29T10:42:41.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.243+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 >> v1:172.21.15.169:6789/0 conn(0x7f3bf013f650 legacy=0x7f3bf006e710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 >> v1:172.21.15.151:6789/0 conn(0x7f3bf00a9d70 legacy=0x7f3bf006e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 1 monclient: found mon.c 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf01366d0 con 0x7f3bf01457f0 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:41.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 client.?.objecter ms_handle_connect 0x7f3bf00a9d70 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 10 monclient: _renew_subs 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf0070330 con 0x7f3bf01457f0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 10 monclient: _renew_subs 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf0071200 con 0x7f3bf01457f0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3bf57cf640 1 librados: init done 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3be00020b0 con 0x7f3bf01457f0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3be77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.245+0000 7f3becff9640 4 set_mon_vals no callback set 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.246+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f3be0003c20 con 0x7f3bf01457f0 2024-03-29T10:42:41.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.246+0000 7f3be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.246+0000 7f3be77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.246+0000 7f3be77fe640 10 monclient: dump: 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.248+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f3be0003d80 con 0x7f3bf01457f0 2024-03-29T10:42:41.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.249+0000 7f3beeffd640 10 monclient: get_auth_request con 0x7f3bd006dbd0 auth_method 0 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.249+0000 7f3be77fe640 10 client.4439.objecter ms_handle_connect 0x7f3bd006dbd0 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.249+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f3be0084df0 con 0x7f3bf01457f0 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.250+0000 7f3be77fe640 10 client.4439.objecter ms_dispatch 0x7f3bf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.250+0000 7f3be77fe640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.250+0000 7f3be77fe640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:41.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.250+0000 7f3bf57cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.250+0000 7f3be77fe640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T10:42:41.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.251+0000 7f3bf57cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.252+0000 7f3bf57cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.237 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.252+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd4005080 con 0x7f3bf01457f0 2024-03-29T10:42:41.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.260+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3be0050500 con 0x7f3bf01457f0 2024-03-29T10:42:41.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.260+0000 7f3be77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.244 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.260+0000 7f3be77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:41.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.583+0000 7f3bf57cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.583+0000 7f3bf57cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.583+0000 7f3bf57cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.583+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bd4005330 con 0x7f3bf01457f0 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.584+0000 7f3be77fe640 1 -- 172.21.15.151:0/43406365 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f3be0052600 con 0x7f3bf01457f0 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.584+0000 7f3be77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.584+0000 7f3be77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:41.568 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:41.568 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:41.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 10 librados: watch_flush enter 2024-03-29T10:42:41.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 10 librados: watch_flush exit 2024-03-29T10:42:41.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T10:42:41.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 10 client.4439.objecter successfully canceled tick 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 >> v1:172.21.15.151:6816/1549873000 conn(0x7f3bd006dbd0 legacy=0x7f3bd0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 10 monclient: shutdown 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.588+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 >> v1:172.21.15.151:6790/0 conn(0x7f3bf01457f0 legacy=0x7f3bf01e64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.589+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 shutdown_connections 2024-03-29T10:42:41.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.589+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 >> 172.21.15.151:0/43406365 conn(0x7f3bf0066a00 msgr2=0x7f3bf013dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:41.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.592+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 shutdown_connections 2024-03-29T10:42:41.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.592+0000 7f3bf57cf640 1 -- 172.21.15.151:0/43406365 wait complete. 2024-03-29T10:42:41.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:41.593+0000 7f3bf57cf640 1 librados: shutdown 2024-03-29T10:42:41.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:48.403 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.574+0000 7ff6f732e640 10 monclient: get_monmap_and_config 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.574+0000 7ff6f732e640 10 monclient: build_initial_monmap 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.574+0000 7ff6f732e640 10 monclient: monmap: 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:48.575404+0000 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:48.575404+0000 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:48.558 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.575+0000 7ff6f732e640 1 Processor -- start 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.575+0000 7ff6f732e640 1 -- start start 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.575+0000 7ff6f732e640 10 monclient: init 2024-03-29T10:42:48.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.576+0000 7ff6f732e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:48.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.576+0000 7ff6f732e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.576+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e8006890 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e8006c70 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e802b4c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802c830 con 0x7ff6e802b4c0 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802afa0 con 0x7ff6e8006890 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802aba0 con 0x7ff6e8006c70 2024-03-29T10:42:48.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:48.563 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.579+0000 7ff6f732e640 10 monclient(hunting): authenticate will time out at 1299.748874s 2024-03-29T10:42:48.563 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.580+0000 7ff6f5b2b640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7ff6e8006c70 0x7ff6e80c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45592/0 (socket says 172.21.15.151:45592) 2024-03-29T10:42:48.563 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.580+0000 7ff6f5b2b640 1 -- 172.21.15.151:0/2359606229 learned_addr learned my addr 172.21.15.151:0/2359606229 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.582+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009444422 0 0) 0x7ff6e802afa0 con 0x7ff6e8006890 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.583+0000 7ff6f532a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.583+0000 7ff6f532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.583+0000 7ff6f532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.583+0000 7ff6f532a640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:42:48.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.583+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc004630 con 0x7ff6e8006890 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345008301 0 0) 0x7ff6e802c830 con 0x7ff6e802b4c0 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e802afa0 con 0x7ff6e802b4c0 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263858300 0 0) 0x7ff6e802aba0 con 0x7ff6e8006c70 2024-03-29T10:42:48.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e802c830 con 0x7ff6e8006c70 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866580548 0 0) 0x7ff6e802c830 con 0x7ff6e8006c70 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 >> v1:172.21.15.169:6789/0 conn(0x7ff6e8006890 legacy=0x7ff6e80c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 >> v1:172.21.15.151:6789/0 conn(0x7ff6e802b4c0 legacy=0x7ff6e80cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 monclient: found mon.noname-b 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.584+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e802ad30 con 0x7ff6e8006c70 2024-03-29T10:42:48.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: _finish_auth 0 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6e0004de0 con 0x7ff6e8006c70 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: dump: 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.571 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e0001080 con 0x7ff6e8006c70 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 1 -- 172.21.15.151:0/2359606229 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6e0001390 con 0x7ff6e8006c70 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.585+0000 7ff6f532a640 10 monclient: dump: 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.572 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.573 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:48.573 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:48.573 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:48.573 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.586+0000 7ff6f732e640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.587+0000 7ff6f732e640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.587+0000 7ff6f732e640 4 set_mon_vals no callback set 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.588+0000 7ff6f732e640 10 monclient: shutdown 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.588+0000 7ff6f732e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.588+0000 7ff6f732e640 1 -- 172.21.15.151:0/2359606229 >> v1:172.21.15.151:6790/0 conn(0x7ff6e8006c70 legacy=0x7ff6e80c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.590+0000 7ff6f732e640 1 -- 172.21.15.151:0/2359606229 shutdown_connections 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.590+0000 7ff6f732e640 1 -- 172.21.15.151:0/2359606229 >> 172.21.15.151:0/2359606229 conn(0x7ff6e80058f0 msgr2=0x7ff6e8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:48.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.590+0000 7ff6f732e640 1 -- 172.21.15.151:0/2359606229 shutdown_connections 2024-03-29T10:42:48.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.590+0000 7ff6f732e640 1 -- 172.21.15.151:0/2359606229 wait complete. 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.591+0000 7ff6f732e640 10 monclient: build_initial_monmap 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.592+0000 7ff6f732e640 10 monclient: monmap: 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:48.576 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.592+0000 7ff6f732e640 1 librados: starting msgr at 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.592+0000 7ff6f732e640 1 librados: starting objecter 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.592+0000 7ff6f732e640 1 Processor -- start 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.593+0000 7ff6f732e640 1 -- start start 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.593+0000 7ff6f732e640 1 librados: setting wanted keys 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.593+0000 7ff6f732e640 1 librados: calling monclient init 2024-03-29T10:42:48.577 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.593+0000 7ff6f732e640 10 monclient: init 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.594+0000 7ff6f732e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.594+0000 7ff6f732e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.594+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e8006890 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.594+0000 7ff6f632c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7ff6e8006890 0x7ff6e802d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:51040/0 (socket says 172.21.15.151:51040) 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.594+0000 7ff6f632c640 1 -- 172.21.15.151:0/775024994 learned_addr learned my addr 172.21.15.151:0/775024994 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e802b4c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e8031060 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802eb80 con 0x7ff6e802b4c0 2024-03-29T10:42:48.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802d0a0 con 0x7ff6e8031060 2024-03-29T10:42:48.579 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e802ccc0 con 0x7ff6e8006890 2024-03-29T10:42:48.579 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:48.579 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.595+0000 7ff6f732e640 10 monclient(hunting): authenticate will time out at 1299.764880s 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 client.?.objecter ms_handle_connect 0x7ff6e8006890 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395445815 0 0) 0x7ff6e802ccc0 con 0x7ff6e8006890 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4004eb0 con 0x7ff6e8006890 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2540901800 0 0) 0x7ff6d4004eb0 con 0x7ff6e8006890 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e802ccc0 con 0x7ff6e8006890 2024-03-29T10:42:48.583 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6ec001910 con 0x7ff6e8006890 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.596+0000 7ff6e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:48.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.597+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3579122893 0 0) 0x7ff6e802ccc0 con 0x7ff6e8006890 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.597+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 >> v1:172.21.15.169:6789/0 conn(0x7ff6e8031060 legacy=0x7ff6e80302e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 >> v1:172.21.15.151:6789/0 conn(0x7ff6e802b4c0 legacy=0x7ff6e815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 1 monclient: found mon.c 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e802ce50 con 0x7ff6e8006890 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 10 monclient: _renew_subs 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e80c4820 con 0x7ff6e8006890 2024-03-29T10:42:48.585 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 10 monclient: _renew_subs 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e80309f0 con 0x7ff6e8006890 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f732e640 1 librados: init done 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6ec001d70 con 0x7ff6e8006890 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff6ec003a60 con 0x7ff6e8006890 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6e6ffd640 10 monclient: dump: 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.586 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:48.587 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.598+0000 7ff6f4b29640 4 set_mon_vals no callback set 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.600+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7ff6ec003c40 con 0x7ff6e8006890 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7ff6ec084b70 con 0x7ff6e8006890 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 10 client.4445.objecter ms_dispatch 0x7ff6f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6f5b2b640 10 monclient: get_auth_request con 0x7ff6d406dbc0 auth_method 0 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6f732e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6f732e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6f732e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8005080 con 0x7ff6e8006890 2024-03-29T10:42:48.588 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.601+0000 7ff6e6ffd640 10 client.4445.objecter ms_handle_connect 0x7ff6d406dbc0 2024-03-29T10:42:48.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.610+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6ec050450 con 0x7ff6e8006890 2024-03-29T10:42:48.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.610+0000 7ff6e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.593 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.610+0000 7ff6e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:48.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.931+0000 7ff6f732e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.931+0000 7ff6f732e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.931+0000 7ff6f732e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:42:48.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.931+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d8005330 con 0x7ff6e8006890 2024-03-29T10:42:48.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.934+0000 7ff6e6ffd640 1 -- 172.21.15.151:0/775024994 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7ff6ec052380 con 0x7ff6e8006890 2024-03-29T10:42:48.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.934+0000 7ff6e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.934+0000 7ff6e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:48.918 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:48.918 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.938+0000 7ff6f732e640 10 librados: watch_flush enter 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.938+0000 7ff6f732e640 10 librados: watch_flush exit 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.938+0000 7ff6f732e640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.938+0000 7ff6f732e640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 >> v1:172.21.15.151:6816/1549873000 conn(0x7ff6d406dbc0 legacy=0x7ff6d4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 10 monclient: shutdown 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 >> v1:172.21.15.151:6790/0 conn(0x7ff6e8006890 legacy=0x7ff6e802d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 shutdown_connections 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 >> 172.21.15.151:0/775024994 conn(0x7ff6e80058f0 msgr2=0x7ff6e8038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:48.923 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 shutdown_connections 2024-03-29T10:42:48.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 -- 172.21.15.151:0/775024994 wait complete. 2024-03-29T10:42:48.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:48.939+0000 7ff6f732e640 1 librados: shutdown 2024-03-29T10:42:48.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:55.747 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:55.899 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.916+0000 7f82853ad640 10 monclient: get_monmap_and_config 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.916+0000 7f82853ad640 10 monclient: build_initial_monmap 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.916+0000 7f82853ad640 10 monclient: monmap: 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:42:55.917526+0000 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:42:55.917526+0000 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:42:55.900 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.917+0000 7f82853ad640 1 Processor -- start 2024-03-29T10:42:55.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.917+0000 7f82853ad640 1 -- start start 2024-03-29T10:42:55.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.917+0000 7f82853ad640 10 monclient: init 2024-03-29T10:42:55.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.918+0000 7f82853ad640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:55.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.918+0000 7f82853ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:55.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.919+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-c con 0x7f82800a9750 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.919+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-b con 0x7f82800aad20 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-a con 0x7f82800a9b30 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800af4d0 con 0x7f82800a9b30 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800af7b0 con 0x7f82800a9750 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800acab0 con 0x7f82800aad20 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:55.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.920+0000 7f82853ad640 10 monclient(hunting): authenticate will time out at 1307.089595s 2024-03-29T10:42:55.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.922+0000 7f827f7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f82800aad20 0x7f8280141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:48159/0 (socket says 172.21.15.151:48159) 2024-03-29T10:42:55.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.922+0000 7f827f7fe640 1 -- 172.21.15.151:0/2578553088 learned_addr learned my addr 172.21.15.151:0/2578553088 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162062137 0 0) 0x7f82800acab0 con 0x7f82800aad20 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268003520 con 0x7f82800aad20 2024-03-29T10:42:55.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4050932664 0 0) 0x7f8268003520 con 0x7f82800aad20 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 >> v1:172.21.15.169:6789/0 conn(0x7f82800a9750 legacy=0x7f82800ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 >> v1:172.21.15.151:6789/0 conn(0x7f82800a9b30 legacy=0x7f8280149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 monclient: found mon.noname-b 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800aba30 con 0x7f82800aad20 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: _finish_auth 0 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8274001690 con 0x7f82800aad20 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:55.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: dump: 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82740017f0 con 0x7f82800aad20 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 1 -- 172.21.15.151:0/2578553088 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8274001c40 con 0x7f82800aad20 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.925+0000 7f827effd640 10 monclient: dump: 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.926+0000 7f82853ad640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:42:55.913 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.926+0000 7f82853ad640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.926+0000 7f82853ad640 4 set_mon_vals no callback set 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.927+0000 7f82853ad640 10 monclient: shutdown 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.927+0000 7f82853ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.927+0000 7f82853ad640 1 -- 172.21.15.151:0/2578553088 >> v1:172.21.15.151:6790/0 conn(0x7f82800aad20 legacy=0x7f8280141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.927+0000 7f82853ad640 1 -- 172.21.15.151:0/2578553088 shutdown_connections 2024-03-29T10:42:55.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.927+0000 7f82853ad640 1 -- 172.21.15.151:0/2578553088 >> 172.21.15.151:0/2578553088 conn(0x7f82800673e0 msgr2=0x7f82800b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:55.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.928+0000 7f82853ad640 1 -- 172.21.15.151:0/2578553088 shutdown_connections 2024-03-29T10:42:55.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.930+0000 7f82853ad640 1 -- 172.21.15.151:0/2578553088 wait complete. 2024-03-29T10:42:55.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.931+0000 7f82853ad640 10 monclient: build_initial_monmap 2024-03-29T10:42:55.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.931+0000 7f82853ad640 10 monclient: monmap: 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.931+0000 7f82853ad640 1 librados: starting msgr at 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.931+0000 7f82853ad640 1 librados: starting objecter 2024-03-29T10:42:55.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.931+0000 7f82853ad640 1 Processor -- start 2024-03-29T10:42:55.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.932+0000 7f82853ad640 1 -- start start 2024-03-29T10:42:55.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.932+0000 7f82853ad640 1 librados: setting wanted keys 2024-03-29T10:42:55.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.932+0000 7f82853ad640 1 librados: calling monclient init 2024-03-29T10:42:55.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.932+0000 7f82853ad640 10 monclient: init 2024-03-29T10:42:55.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.932+0000 7f82853ad640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:55.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.933+0000 7f82853ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:55.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.933+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-c con 0x7f82800a9750 addr v1:172.21.15.151:6790/0 2024-03-29T10:42:55.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-a con 0x7f82800a9b30 addr v1:172.21.15.151:6789/0 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 10 monclient(hunting): picked mon.noname-b con 0x7f82800aad20 addr v1:172.21.15.169:6789/0 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828013b080 con 0x7f82800a9b30 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828014d760 con 0x7f82800aad20 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828014e370 con 0x7f82800a9750 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:55.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.936+0000 7f82853ad640 10 monclient(hunting): authenticate will time out at 1307.106367s 2024-03-29T10:42:55.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f82800a9b30 0x7f82801360f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35921/0 (socket says 172.21.15.151:35921) 2024-03-29T10:42:55.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827f7fe640 1 -- 172.21.15.151:0/3470497560 learned_addr learned my addr 172.21.15.151:0/3470497560 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:42:55.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 client.?.objecter ms_handle_connect 0x7f82800a9b30 2024-03-29T10:42:55.924 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812199025 0 0) 0x7f828013b080 con 0x7f82800a9b30 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.940+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c004eb0 con 0x7f82800a9b30 2024-03-29T10:42:55.931 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 client.?.objecter ms_handle_connect 0x7f82800aad20 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 client.?.objecter ms_handle_connect 0x7f82800a9750 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470321397 0 0) 0x7f828014e370 con 0x7f82800a9750 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.941+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f828013b080 con 0x7f82800a9750 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 239765691 0 0) 0x7f826c004eb0 con 0x7f82800a9b30 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f828014e370 con 0x7f82800a9b30 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8274001b00 con 0x7f82800a9b30 2024-03-29T10:42:55.932 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 10 monclient(hunting): dump: 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:55.933 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955040416 0 0) 0x7f828014e370 con 0x7f82800a9b30 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.942+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 >> v1:172.21.15.151:6790/0 conn(0x7f82800a9750 legacy=0x7f828014dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 >> v1:172.21.15.169:6789/0 conn(0x7f82800aad20 legacy=0x7f82801e6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 1 monclient: found mon.a 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280137320 con 0x7f82800a9b30 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 10 monclient: _finish_auth 0 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:42:55.934 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 5 monclient: authenticate success, global_id 4425 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 10 monclient: _renew_subs 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82801de850 con 0x7f82800a9b30 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 10 monclient: _renew_subs 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8280071970 con 0x7f82800a9b30 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f82853ad640 1 librados: init done 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8274001c60 con 0x7f82800a9b30 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.943+0000 7f827dffb640 4 set_mon_vals no callback set 2024-03-29T10:42:55.935 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.944+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8274001270 con 0x7f82800a9b30 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.944+0000 7f827cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.944+0000 7f827cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.944+0000 7f827cff9640 10 monclient: dump: 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:42:55.936 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.945+0000 7f82853ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.945+0000 7f82853ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.945+0000 7f82853ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.945+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8260005080 con 0x7f82800a9b30 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.946+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f8274004c30 con 0x7f82800a9b30 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827ffff640 10 monclient: get_auth_request con 0x7f826c06dd40 auth_method 0 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f8274084aa0 con 0x7f82800a9b30 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 10 client.4425.objecter ms_dispatch 0x7f8280000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T10:42:55.937 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.947+0000 7f827cff9640 10 client.4425.objecter ms_handle_connect 0x7f826c06dd40 2024-03-29T10:42:55.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.955+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8274050380 con 0x7f82800a9b30 2024-03-29T10:42:55.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.955+0000 7f827cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.938 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:55.955+0000 7f827cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:56.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.290+0000 7f82853ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.290+0000 7f82853ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.290+0000 7f82853ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:42:56.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.290+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8260005330 con 0x7f82800a9b30 2024-03-29T10:42:56.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.291+0000 7f827cff9640 1 -- 172.21.15.151:0/3470497560 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f82740522b0 con 0x7f82800a9b30 2024-03-29T10:42:56.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.291+0000 7f827cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.291+0000 7f827cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:56.275 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:42:56.276 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:56.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 10 librados: watch_flush enter 2024-03-29T10:42:56.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 10 librados: watch_flush exit 2024-03-29T10:42:56.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T10:42:56.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 10 client.4425.objecter successfully canceled tick 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 >> v1:172.21.15.151:6816/1549873000 conn(0x7f826c06dd40 legacy=0x7f826c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 10 monclient: shutdown 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 >> v1:172.21.15.151:6789/0 conn(0x7f82800a9b30 legacy=0x7f82801360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 shutdown_connections 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 >> 172.21.15.151:0/3470497560 conn(0x7f82800673e0 msgr2=0x7f82800ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 shutdown_connections 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.296+0000 7f82853ad640 1 -- 172.21.15.151:0/3470497560 wait complete. 2024-03-29T10:42:56.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:42:56.297+0000 7f82853ad640 1 librados: shutdown 2024-03-29T10:42:56.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:03.107 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.275+0000 7f23a9d43640 10 monclient: get_monmap_and_config 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.275+0000 7f23a9d43640 10 monclient: build_initial_monmap 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.275+0000 7f23a9d43640 10 monclient: monmap: 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:03.259 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:03.275983+0000 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:03.275983+0000 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.260 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.277+0000 7f23a9d43640 1 Processor -- start 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 1 -- start start 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 10 monclient: init 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-b con 0x7f239c034850 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.278+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-a con 0x7f239c02a640 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:03.266 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-c con 0x7f239c0327d0 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c02c820 con 0x7f239c02a640 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c02a120 con 0x7f239c0327d0 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0c75b0 con 0x7f239c034850 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.279+0000 7f23a9d43640 10 monclient(hunting): authenticate will time out at 1314.448563s 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.283+0000 7f23a9542640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f239c0327d0 0x7f239c0cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:52155/0 (socket says 172.21.15.151:52155) 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.283+0000 7f23a9542640 1 -- 172.21.15.151:0/493283616 learned_addr learned my addr 172.21.15.151:0/493283616 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:03.267 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840356408 0 0) 0x7f239c02a120 con 0x7f239c0327d0 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394003520 con 0x7f239c0327d0 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655195417 0 0) 0x7f2394003520 con 0x7f239c0327d0 2024-03-29T10:43:03.268 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.284+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 >> v1:172.21.15.151:6790/0 conn(0x7f239c034850 legacy=0x7f239c0c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 >> v1:172.21.15.151:6789/0 conn(0x7f239c02a640 legacy=0x7f239c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 1 monclient: found mon.noname-c 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.285+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c033110 con 0x7f239c0327d0 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:03.270 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f23a404f730 con 0x7f239c0327d0 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: dump: 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:03.271 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:03.272 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:03.272 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a9d43640 5 monclient: authenticate success, global_id 4444 2024-03-29T10:43:03.272 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a9d43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:03.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a4075f10 con 0x7f239c0327d0 2024-03-29T10:43:03.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:03.273 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 1 -- 172.21.15.151:0/493283616 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f23a40758b0 con 0x7f239c0327d0 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a37fe640 10 monclient: dump: 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:03.274 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a9d43640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.286+0000 7f23a9d43640 4 set_mon_vals no callback set 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.287+0000 7f23a9d43640 10 monclient: shutdown 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.287+0000 7f23a9d43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.287+0000 7f23a9d43640 1 -- 172.21.15.151:0/493283616 >> v1:172.21.15.169:6789/0 conn(0x7f239c0327d0 legacy=0x7f239c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.290+0000 7f23a9d43640 1 -- 172.21.15.151:0/493283616 shutdown_connections 2024-03-29T10:43:03.275 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.290+0000 7f23a9d43640 1 -- 172.21.15.151:0/493283616 >> 172.21.15.151:0/493283616 conn(0x7f239c0058f0 msgr2=0x7f239c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:03.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.291+0000 7f23a9d43640 1 -- 172.21.15.151:0/493283616 shutdown_connections 2024-03-29T10:43:03.277 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.291+0000 7f23a9d43640 1 -- 172.21.15.151:0/493283616 wait complete. 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 10 monclient: build_initial_monmap 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 10 monclient: monmap: 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:03.278 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 librados: starting msgr at 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 librados: starting objecter 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 Processor -- start 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 -- start start 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 librados: setting wanted keys 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 1 librados: calling monclient init 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.292+0000 7f23a9d43640 10 monclient: init 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.293+0000 7f23a9d43640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:03.279 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.293+0000 7f23a9d43640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.293+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-b con 0x7f239c02a640 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.293+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-a con 0x7f239c0327d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 10 monclient(hunting): picked mon.noname-c con 0x7f239c034850 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c02e430 con 0x7f239c0327d0 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c030820 con 0x7f239c02a640 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c030a90 con 0x7f239c034850 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:03.280 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.295+0000 7f23a9d43640 10 monclient(hunting): authenticate will time out at 1314.464709s 2024-03-29T10:43:03.286 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.298+0000 7f23a9542640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f239c034850 0x7f239c189490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60407/0 (socket says 172.21.15.151:60407) 2024-03-29T10:43:03.286 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.298+0000 7f23a9542640 1 -- 172.21.15.151:0/2292608704 learned_addr learned my addr 172.21.15.151:0/2292608704 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 client.?.objecter ms_handle_connect 0x7f239c034850 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789014554 0 0) 0x7f239c030a90 con 0x7f239c034850 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.299+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2388004eb0 con 0x7f239c034850 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132897455 0 0) 0x7f2388004eb0 con 0x7f239c034850 2024-03-29T10:43:03.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f239c030a90 con 0x7f239c034850 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f23a404f400 con 0x7f239c034850 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 10 monclient(hunting): dump: 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.288 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.301+0000 7f23a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1169075611 0 0) 0x7f239c030a90 con 0x7f239c034850 2024-03-29T10:43:03.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 >> v1:172.21.15.169:6789/0 conn(0x7f239c02a640 legacy=0x7f239c031490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 >> v1:172.21.15.151:6789/0 conn(0x7f239c0327d0 legacy=0x7f239c1822c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 1 monclient: found mon.c 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0c01c0 con 0x7f239c034850 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a9d43640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a9d43640 10 monclient: _renew_subs 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a9d43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:03.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.302+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c0c0410 con 0x7f239c034850 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.303+0000 7f23a9d43640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.303+0000 7f23a9d43640 10 monclient: _renew_subs 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.303+0000 7f23a9d43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.303+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c182bb0 con 0x7f239c034850 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.303+0000 7f23a9d43640 1 librados: init done 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.305+0000 7f23a9d43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.305+0000 7f23a9d43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.305+0000 7f23a9d43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.305+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c182e50 con 0x7f239c034850 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a404fdd0 con 0x7f239c034850 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f23a4067bf0 con 0x7f239c034850 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:03.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 10 monclient: dump: 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.307+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f23a4067e10 con 0x7f239c034850 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f23a40ee640 con 0x7f239c034850 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a17fa640 10 client.4457.objecter ms_dispatch 0x7f23a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:03.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a17fa640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:03.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a17fa640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:03.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a17fa640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T10:43:03.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.308+0000 7f23a27fc640 4 set_mon_vals no callback set 2024-03-29T10:43:03.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.309+0000 7f23a8d41640 10 monclient: get_auth_request con 0x7f238806db90 auth_method 0 2024-03-29T10:43:03.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.310+0000 7f23a17fa640 10 client.4457.objecter ms_handle_connect 0x7f238806db90 2024-03-29T10:43:03.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.317+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23a40b9f60 con 0x7f239c034850 2024-03-29T10:43:03.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.317+0000 7f23a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.317+0000 7f23a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:03.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.645+0000 7f23a9d43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.645+0000 7f23a9d43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.645+0000 7f23a9d43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:03.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.645+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239c004cd0 con 0x7f239c034850 2024-03-29T10:43:03.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.647+0000 7f23a17fa640 1 -- 172.21.15.151:0/2292608704 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f23a40bbe50 con 0x7f239c034850 2024-03-29T10:43:03.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.647+0000 7f23a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.647+0000 7f23a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:03.633 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:03.633 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.653+0000 7f23a9d43640 10 librados: watch_flush enter 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.653+0000 7f23a9d43640 10 librados: watch_flush exit 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.653+0000 7f23a9d43640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.653+0000 7f23a9d43640 10 client.4457.objecter successfully canceled tick 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 >> v1:172.21.15.151:6816/1549873000 conn(0x7f238806db90 legacy=0x7f2388070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 10 monclient: shutdown 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 >> v1:172.21.15.151:6790/0 conn(0x7f239c034850 legacy=0x7f239c189490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 shutdown_connections 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.654+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 >> 172.21.15.151:0/2292608704 conn(0x7f239c0058f0 msgr2=0x7f239c029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.655+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 shutdown_connections 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.655+0000 7f23a9d43640 1 -- 172.21.15.151:0/2292608704 wait complete. 2024-03-29T10:43:03.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:03.655+0000 7f23a9d43640 1 librados: shutdown 2024-03-29T10:43:03.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:10.469 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:10.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.629+0000 7f03764f1640 10 monclient: get_monmap_and_config 2024-03-29T10:43:10.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.629+0000 7f03764f1640 10 monclient: build_initial_monmap 2024-03-29T10:43:10.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.629+0000 7f03764f1640 10 monclient: monmap: 2024-03-29T10:43:10.612 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:10.612 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:10.630252+0000 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:10.630252+0000 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.630+0000 7f03764f1640 1 Processor -- start 2024-03-29T10:43:10.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.630+0000 7f03764f1640 1 -- start start 2024-03-29T10:43:10.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.630+0000 7f03764f1640 10 monclient: init 2024-03-29T10:43:10.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.631+0000 7f03764f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:10.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.631+0000 7f03764f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.631+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-b con 0x7f03701457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.631+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-a con 0x7f037013f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036ffff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f03701457f0 0x7f037013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56398/0 (socket says 172.21.15.151:56398) 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f0374a8e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f03700a9d70 0x7f037014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:54160/0 (socket says 172.21.15.151:54160) 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036f7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f037013f650 0x7f03700b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:43694/0 (socket says 172.21.15.151:43694) 2024-03-29T10:43:10.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036ffff640 1 -- 172.21.15.151:0/1261823485 learned_addr learned my addr 172.21.15.151:0/1261823485 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-c con 0x7f03700a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700aaef0 con 0x7f037013f650 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0370145bd0 con 0x7f03700a9d70 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0370145eb0 con 0x7f03701457f0 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:10.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251340004 0 0) 0x7f03700aaef0 con 0x7f037013f650 2024-03-29T10:43:10.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f03764f1640 10 monclient(hunting): authenticate will time out at 1321.802336s 2024-03-29T10:43:10.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.632+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0358003520 con 0x7f037013f650 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185694105 0 0) 0x7f0370145bd0 con 0x7f03700a9d70 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03700aaef0 con 0x7f03700a9d70 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220554807 0 0) 0x7f0370145eb0 con 0x7f03701457f0 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:43:10.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0370145bd0 con 0x7f03701457f0 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881806186 0 0) 0x7f0358003520 con 0x7f037013f650 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 >> v1:172.21.15.151:6790/0 conn(0x7f03701457f0 legacy=0x7f037013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 >> v1:172.21.15.169:6789/0 conn(0x7f03700a9d70 legacy=0x7f037014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 monclient: found mon.noname-a 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0370146340 con 0x7f037013f650 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: _finish_auth 0 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0364001690 con 0x7f037013f650 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f03764f1640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f03764f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:10.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f036effd640 10 monclient: dump: 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.633+0000 7f03764f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03640018b0 con 0x7f037013f650 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 1 -- 172.21.15.151:0/1261823485 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0364001c40 con 0x7f037013f650 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f036effd640 10 monclient: dump: 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.623 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f03764f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f03764f1640 4 set_mon_vals no callback set 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f03764f1640 10 monclient: shutdown 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.634+0000 7f03764f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.635+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 >> v1:172.21.15.151:6789/0 conn(0x7f037013f650 legacy=0x7f03700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.635+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 shutdown_connections 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.635+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 >> 172.21.15.151:0/1261823485 conn(0x7f0370066a00 msgr2=0x7f0370066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.635+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 shutdown_connections 2024-03-29T10:43:10.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.635+0000 7f03764f1640 1 -- 172.21.15.151:0/1261823485 wait complete. 2024-03-29T10:43:10.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.636+0000 7f03764f1640 10 monclient: build_initial_monmap 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.636+0000 7f03764f1640 10 monclient: monmap: 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 librados: starting msgr at 2024-03-29T10:43:10.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 librados: starting objecter 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 Processor -- start 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 -- start start 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 librados: setting wanted keys 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 1 librados: calling monclient init 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.637+0000 7f03764f1640 10 monclient: init 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.638+0000 7f03764f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.638+0000 7f03764f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.638+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-a con 0x7f03700a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036ffff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f03700a9d70 0x7f03701d7180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:44376/0 (socket says 172.21.15.151:44376) 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036ffff640 1 -- 172.21.15.151:0/1884574471 learned_addr learned my addr 172.21.15.151:0/1884574471 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-b con 0x7f037013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 10 monclient(hunting): picked mon.noname-c con 0x7f03701457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700adce0 con 0x7f03700a9d70 2024-03-29T10:43:10.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700af450 con 0x7f037013f650 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700ad920 con 0x7f03701457f0 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f03764f1640 10 monclient(hunting): authenticate will time out at 1321.808786s 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 client.?.objecter ms_handle_connect 0x7f03700a9d70 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297834504 0 0) 0x7f03700adce0 con 0x7f03700a9d70 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.639+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c004eb0 con 0x7f03700a9d70 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 client.?.objecter ms_handle_connect 0x7f03701457f0 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:10.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700620033 0 0) 0x7f035c004eb0 con 0x7f03700a9d70 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03700adce0 con 0x7f03700a9d70 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0360001870 con 0x7f03700a9d70 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:10.634 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 client.?.objecter ms_handle_connect 0x7f037013f650 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4164888286 0 0) 0x7f03700adce0 con 0x7f03700a9d70 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 >> v1:172.21.15.151:6790/0 conn(0x7f03701457f0 legacy=0x7f03701e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 >> v1:172.21.15.169:6789/0 conn(0x7f037013f650 legacy=0x7f03700b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 monclient: found mon.a 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03701d7890 con 0x7f03700a9d70 2024-03-29T10:43:10.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.640+0000 7f036cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 10 monclient: _renew_subs 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03700ae170 con 0x7f03700a9d70 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03600019d0 con 0x7f03700a9d70 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0360004a40 con 0x7f03700a9d70 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036cff9640 10 monclient: dump: 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:10.636 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f036dffb640 4 set_mon_vals no callback set 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 10 monclient: _renew_subs 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0370142240 con 0x7f03700a9d70 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.641+0000 7f03764f1640 1 librados: init done 2024-03-29T10:43:10.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.642+0000 7f03764f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.643+0000 7f03764f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.643+0000 7f03764f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.643+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03700adab0 con 0x7f03700a9d70 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.643+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f0360004ba0 con 0x7f03700a9d70 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.644+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f0360084b60 con 0x7f03700a9d70 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.644+0000 7f036f7fe640 10 monclient: get_auth_request con 0x7f035c0764e0 auth_method 0 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.645+0000 7f036cff9640 10 client.4431.objecter ms_dispatch 0x7f0370000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.645+0000 7f036cff9640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.645+0000 7f036cff9640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.645+0000 7f036cff9640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T10:43:10.638 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.645+0000 7f036cff9640 10 client.4431.objecter ms_handle_connect 0x7f035c0764e0 2024-03-29T10:43:10.639 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.654+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03700adab0 con 0x7f03700a9d70 2024-03-29T10:43:10.639 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.654+0000 7f036cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.639 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.654+0000 7f036cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:10.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.992+0000 7f03764f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.992+0000 7f03764f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.992+0000 7f03764f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:43:10.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.992+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03700adab0 con 0x7f03700a9d70 2024-03-29T10:43:10.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.994+0000 7f036cff9640 1 -- 172.21.15.151:0/1884574471 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f03700adab0 con 0x7f03700a9d70 2024-03-29T10:43:10.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.994+0000 7f036cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.994+0000 7f036cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:10.977 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:10.977 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:10.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.998+0000 7f03764f1640 10 librados: watch_flush enter 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 10 librados: watch_flush exit 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 10 client.4431.objecter successfully canceled tick 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 >> v1:172.21.15.151:6816/1549873000 conn(0x7f035c0764e0 legacy=0x7f035c0789a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 10 monclient: shutdown 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 >> v1:172.21.15.151:6789/0 conn(0x7f03700a9d70 legacy=0x7f03701d7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 shutdown_connections 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 >> 172.21.15.151:0/1884574471 conn(0x7f0370066a00 msgr2=0x7f03700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 shutdown_connections 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:10.999+0000 7f03764f1640 1 -- 172.21.15.151:0/1884574471 wait complete. 2024-03-29T10:43:10.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:11.000+0000 7f03764f1640 1 librados: shutdown 2024-03-29T10:43:11.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:17.807 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.977+0000 7f469b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.977+0000 7f469b59e640 10 monclient: build_initial_monmap 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.977+0000 7f469b59e640 10 monclient: monmap: 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:17.978687+0000 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:17.978687+0000 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:17.961 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:17.962 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:17.962 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.978+0000 7f469b59e640 1 Processor -- start 2024-03-29T10:43:17.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.978+0000 7f469b59e640 1 -- start start 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.978+0000 7f469b59e640 10 monclient: init 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f469c0b4820 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f469c0a9ce0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f469c0b3cd0 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0b4180 con 0x7f469c0a9ce0 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0aaf60 con 0x7f469c0b3cd0 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c13c200 con 0x7f469c0b4820 2024-03-29T10:43:17.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:17.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.980+0000 7f469b59e640 10 monclient(hunting): authenticate will time out at 1329.149649s 2024-03-29T10:43:17.966 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.983+0000 7f4699d9b640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f469c0a9ce0 0x7f469c145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:47049/0 (socket says 172.21.15.151:47049) 2024-03-29T10:43:17.966 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.983+0000 7f4699d9b640 1 -- 172.21.15.151:0/644069293 learned_addr learned my addr 172.21.15.151:0/644069293 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:17.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241528105 0 0) 0x7f4694001490 con 0x7f469c0a9ce0 2024-03-29T10:43:17.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:43:17.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.986+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4684003520 con 0x7f469c0a9ce0 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179972517 0 0) 0x7f469c0aaf60 con 0x7f469c0b3cd0 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694001490 con 0x7f469c0b3cd0 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836991300 0 0) 0x7f4684003520 con 0x7f469c0a9ce0 2024-03-29T10:43:17.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 >> v1:172.21.15.151:6790/0 conn(0x7f469c0b4820 legacy=0x7f469c13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 >> v1:172.21.15.169:6789/0 conn(0x7f469c0b3cd0 legacy=0x7f469c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 monclient: found mon.noname-a 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c145e80 con 0x7f469c0a9ce0 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: _finish_auth 0 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469b59e640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4694001a00 con 0x7f469c0a9ce0 2024-03-29T10:43:17.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: dump: 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4694001be0 con 0x7f469c0a9ce0 2024-03-29T10:43:17.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 1 -- 172.21.15.151:0/644069293 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4694001fb0 con 0x7f469c0a9ce0 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469959a640 10 monclient: dump: 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.975 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.987+0000 7f469b59e640 4 set_mon_vals no callback set 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.988+0000 7f469b59e640 10 monclient: shutdown 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.988+0000 7f469b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.988+0000 7f469b59e640 1 -- 172.21.15.151:0/644069293 >> v1:172.21.15.151:6789/0 conn(0x7f469c0a9ce0 legacy=0x7f469c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.989+0000 7f469b59e640 1 -- 172.21.15.151:0/644069293 shutdown_connections 2024-03-29T10:43:17.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.989+0000 7f469b59e640 1 -- 172.21.15.151:0/644069293 >> 172.21.15.151:0/644069293 conn(0x7f469c06d070 msgr2=0x7f469c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.989+0000 7f469b59e640 1 -- 172.21.15.151:0/644069293 shutdown_connections 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.990+0000 7f469b59e640 1 -- 172.21.15.151:0/644069293 wait complete. 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 10 monclient: build_initial_monmap 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 10 monclient: monmap: 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:17.983 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 librados: starting msgr at 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 librados: starting objecter 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 Processor -- start 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 -- start start 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 librados: setting wanted keys 2024-03-29T10:43:17.984 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 1 librados: calling monclient init 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.992+0000 7f469b59e640 10 monclient: init 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f469c0a9ce0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469a59c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f469c0a9ce0 0x7f469c1db510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:42304/0 (socket says 172.21.15.151:42304) 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469a59c640 1 -- 172.21.15.151:0/256753849 learned_addr learned my addr 172.21.15.151:0/256753849 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f469c0b3cd0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f469c0b4820 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c06e370 con 0x7f469c0b3cd0 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c13f690 con 0x7f469c0b4820 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c06f900 con 0x7f469c0a9ce0 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f469b59e640 10 monclient(hunting): authenticate will time out at 1329.164066s 2024-03-29T10:43:17.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f468affd640 10 client.?.objecter ms_handle_connect 0x7f469c0a9ce0 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f468affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f468affd640 10 client.?.objecter ms_handle_connect 0x7f469c0b3cd0 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.994+0000 7f468affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848452723 0 0) 0x7f469c06f900 con 0x7f469c0a9ce0 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467c004eb0 con 0x7f469c0a9ce0 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 client.?.objecter ms_handle_connect 0x7f469c0b4820 2024-03-29T10:43:17.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471959675 0 0) 0x7f469c06e370 con 0x7f469c0b3cd0 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c06f900 con 0x7f469c0b3cd0 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817861820 0 0) 0x7f469c13f690 con 0x7f469c0b4820 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.995+0000 7f468affd640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c06e370 con 0x7f469c0b4820 2024-03-29T10:43:17.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890296943 0 0) 0x7f467c004eb0 con 0x7f469c0a9ce0 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467c005620 con 0x7f469c0a9ce0 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f468c0017d0 con 0x7f469c0a9ce0 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): dump: 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.991 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711291008 0 0) 0x7f469c06e370 con 0x7f469c0b4820 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467c004eb0 con 0x7f469c0b4820 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4690001930 con 0x7f469c0b4820 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.996+0000 7f468affd640 10 monclient(hunting): dump: 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.992 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486794095 0 0) 0x7f469c06f900 con 0x7f469c0b3cd0 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467c006150 con 0x7f469c0b3cd0 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4694001ef0 con 0x7f469c0b3cd0 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient(hunting): dump: 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.993 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239584961 0 0) 0x7f467c004eb0 con 0x7f469c0b4820 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 >> v1:172.21.15.151:6790/0 conn(0x7f469c0a9ce0 legacy=0x7f469c1db510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 >> v1:172.21.15.151:6789/0 conn(0x7f469c0b3cd0 legacy=0x7f469c1dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 monclient: found mon.b 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c06fbe0 con 0x7f469c0b4820 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient: _finish_auth 0 2024-03-29T10:43:17.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f468affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 10 monclient: _renew_subs 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c06e5f0 con 0x7f469c0b4820 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 10 monclient: _renew_subs 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c0b1340 con 0x7f469c0b4820 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.997+0000 7f469b59e640 1 librados: init done 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46900035f0 con 0x7f469c0b4820 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:17.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4690001930 con 0x7f469c0b4820 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f468affd640 10 monclient: dump: 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:17.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.998+0000 7f4698d99640 4 set_mon_vals no callback set 2024-03-29T10:43:17.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.999+0000 7f469b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.999+0000 7f469b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.999+0000 7f469b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:17.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:17.999+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4680005080 con 0x7f469c0b4820 2024-03-29T10:43:17.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.000+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f46900195d0 con 0x7f469c0b4820 2024-03-29T10:43:17.998 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4690083200 con 0x7f469c0b4820 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f468affd640 10 client.4453.objecter ms_dispatch 0x7f469c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f468affd640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f468affd640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f468affd640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.001+0000 7f469a59c640 10 monclient: get_auth_request con 0x7f467c06e7d0 auth_method 0 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.002+0000 7f468affd640 10 client.4453.objecter ms_handle_connect 0x7f467c06e7d0 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.008+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f469004eb40 con 0x7f469c0b4820 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.008+0000 7f468affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.999 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.008+0000 7f468affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:18.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.387+0000 7f469b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.387+0000 7f469b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.387+0000 7f469b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:18.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.387+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4680005330 con 0x7f469c0b4820 2024-03-29T10:43:18.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.388+0000 7f468affd640 1 -- 172.21.15.151:0/256753849 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4690050a70 con 0x7f469c0b4820 2024-03-29T10:43:18.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.388+0000 7f468affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.388+0000 7f468affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:18.372 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:18.372 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 10 librados: watch_flush enter 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 10 librados: watch_flush exit 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 10 client.4453.objecter successfully canceled tick 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 >> v1:172.21.15.151:6816/1549873000 conn(0x7f467c06e7d0 legacy=0x7f467c070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 10 monclient: shutdown 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 >> v1:172.21.15.169:6789/0 conn(0x7f469c0b4820 legacy=0x7f469c1e5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 shutdown_connections 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 >> 172.21.15.151:0/256753849 conn(0x7f469c06d070 msgr2=0x7f469c0b72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:18.376 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 shutdown_connections 2024-03-29T10:43:18.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 -- 172.21.15.151:0/256753849 wait complete. 2024-03-29T10:43:18.377 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:18.392+0000 7f469b59e640 1 librados: shutdown 2024-03-29T10:43:18.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:25.198 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:25.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.382+0000 7fe254b6d640 10 monclient: get_monmap_and_config 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.382+0000 7fe254b6d640 10 monclient: build_initial_monmap 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.382+0000 7fe254b6d640 10 monclient: monmap: 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:25.383598+0000 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:25.383598+0000 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:25.366 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:25.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.383+0000 7fe254b6d640 1 Processor -- start 2024-03-29T10:43:25.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.384+0000 7fe254b6d640 1 -- start start 2024-03-29T10:43:25.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.384+0000 7fe254b6d640 10 monclient: init 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2501457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe25013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2500a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2500aaef0 con 0x7fe2501457f0 2024-03-29T10:43:25.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe250145bd0 con 0x7fe25013f650 2024-03-29T10:43:25.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe250145eb0 con 0x7fe2500a9d70 2024-03-29T10:43:25.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:25.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.385+0000 7fe254b6d640 10 monclient(hunting): authenticate will time out at 1336.555097s 2024-03-29T10:43:25.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.393+0000 7fe24dd9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe25013f650 0x7fe2500b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:35219/0 (socket says 172.21.15.151:35219) 2024-03-29T10:43:25.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.393+0000 7fe24dd9b640 1 -- 172.21.15.151:0/1287322094 learned_addr learned my addr 172.21.15.151:0/1287322094 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:25.380 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711100250 0 0) 0x7fe250145bd0 con 0x7fe25013f650 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.397+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe23c003520 con 0x7fe25013f650 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792760061 0 0) 0x7fe2500aaef0 con 0x7fe2501457f0 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.381 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:43:25.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.398+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe250145bd0 con 0x7fe2501457f0 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790665200 0 0) 0x7fe250145eb0 con 0x7fe2500a9d70 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2500aaef0 con 0x7fe2500a9d70 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446597854 0 0) 0x7fe250145bd0 con 0x7fe2501457f0 2024-03-29T10:43:25.383 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 >> v1:172.21.15.151:6790/0 conn(0x7fe2500a9d70 legacy=0x7fe25014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 >> v1:172.21.15.169:6789/0 conn(0x7fe25013f650 legacy=0x7fe2500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 monclient: found mon.noname-a 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250146340 con 0x7fe2501457f0 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient: _finish_auth 0 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.399+0000 7fe24d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe238001670 con 0x7fe2501457f0 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 10 monclient: dump: 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:25.384 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe254b6d640 5 monclient: authenticate success, global_id 4440 2024-03-29T10:43:25.385 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe254b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 1 -- 172.21.15.151:0/1287322094 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2380017d0 con 0x7fe2501457f0 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe24d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe254b6d640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.400+0000 7fe254b6d640 4 set_mon_vals no callback set 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.401+0000 7fe254b6d640 10 monclient: shutdown 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.401+0000 7fe254b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:25.386 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.401+0000 7fe254b6d640 1 -- 172.21.15.151:0/1287322094 >> v1:172.21.15.151:6789/0 conn(0x7fe2501457f0 legacy=0x7fe25013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.387 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.403+0000 7fe254b6d640 1 -- 172.21.15.151:0/1287322094 shutdown_connections 2024-03-29T10:43:25.387 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.403+0000 7fe254b6d640 1 -- 172.21.15.151:0/1287322094 >> 172.21.15.151:0/1287322094 conn(0x7fe250066a00 msgr2=0x7fe250066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:25.387 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.404+0000 7fe254b6d640 1 -- 172.21.15.151:0/1287322094 shutdown_connections 2024-03-29T10:43:25.387 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.404+0000 7fe254b6d640 1 -- 172.21.15.151:0/1287322094 wait complete. 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.405+0000 7fe254b6d640 10 monclient: build_initial_monmap 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.405+0000 7fe254b6d640 10 monclient: monmap: 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:25.389 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.405+0000 7fe254b6d640 1 librados: starting msgr at 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.405+0000 7fe254b6d640 1 librados: starting objecter 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.405+0000 7fe254b6d640 1 Processor -- start 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.406+0000 7fe254b6d640 1 -- start start 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.406+0000 7fe254b6d640 1 librados: setting wanted keys 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.406+0000 7fe254b6d640 1 librados: calling monclient init 2024-03-29T10:43:25.390 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.406+0000 7fe254b6d640 10 monclient: init 2024-03-29T10:43:25.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.407+0000 7fe254b6d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:25.391 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.407+0000 7fe254b6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:25.394 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.411+0000 7fe24e59c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe2500a9d70 0x7fe250137cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:49585/0 (socket says 172.21.15.151:49585) 2024-03-29T10:43:25.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.412+0000 7fe24e59c640 1 -- 172.21.15.151:0/202817389 learned_addr learned my addr 172.21.15.151:0/202817389 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:25.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.412+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2500a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:25.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.412+0000 7fe246ffd640 10 client.?.objecter ms_handle_connect 0x7fe2500a9d70 2024-03-29T10:43:25.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.412+0000 7fe246ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:25.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe25013f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:25.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe246ffd640 10 client.?.objecter ms_handle_connect 0x7fe25013f650 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe246ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2501457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2501389d0 con 0x7fe2500a9d70 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe250140ab0 con 0x7fe2501457f0 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe250139a80 con 0x7fe25013f650 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.415+0000 7fe254b6d640 10 monclient(hunting): authenticate will time out at 1336.585442s 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288488860 0 0) 0x7fe250139a80 con 0x7fe25013f650 2024-03-29T10:43:25.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:43:25.404 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe230004eb0 con 0x7fe25013f650 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.420+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4730725 0 0) 0x7fe230004eb0 con 0x7fe25013f650 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe250139a80 con 0x7fe25013f650 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe248001950 con 0x7fe25013f650 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3487359964 0 0) 0x7fe250139a80 con 0x7fe25013f650 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 >> v1:172.21.15.169:6789/0 conn(0x7fe2501457f0 legacy=0x7fe2501dd930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 >> v1:172.21.15.151:6789/0 conn(0x7fe2500a9d70 legacy=0x7fe250137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:25.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 monclient: found mon.c 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.421+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250139d60 con 0x7fe25013f650 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe246ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe246ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 10 monclient: _renew_subs 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25013a040 con 0x7fe25013f650 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 10 monclient: _renew_subs 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:25.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2501447f0 con 0x7fe25013f650 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.422+0000 7fe254b6d640 1 librados: init done 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.424+0000 7fe254b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.424+0000 7fe254b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.424+0000 7fe254b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.424+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250075f10 con 0x7fe25013f650 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.425+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2480047e0 con 0x7fe25013f650 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.425+0000 7fe246ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.425+0000 7fe247fff640 4 set_mon_vals no callback set 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.426+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe248004b70 con 0x7fe25013f650 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.426+0000 7fe246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.426+0000 7fe246ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.426+0000 7fe246ffd640 10 monclient: dump: 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.426+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe248004d90 con 0x7fe25013f650 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.427+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe248084be0 con 0x7fe25013f650 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.427+0000 7fe246ffd640 10 client.4472.objecter ms_dispatch 0x7fe250000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:25.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.428+0000 7fe246ffd640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.428+0000 7fe246ffd640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.428+0000 7fe24e59c640 10 monclient: get_auth_request con 0x7fe23006dc00 auth_method 0 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.428+0000 7fe246ffd640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.428+0000 7fe246ffd640 10 client.4472.objecter ms_handle_connect 0x7fe23006dc00 2024-03-29T10:43:25.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.436+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe248050540 con 0x7fe25013f650 2024-03-29T10:43:25.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.436+0000 7fe246ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.436+0000 7fe246ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:25.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.761+0000 7fe254b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.761+0000 7fe254b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.761+0000 7fe254b6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:25.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.761+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2500b3680 con 0x7fe25013f650 2024-03-29T10:43:25.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.763+0000 7fe246ffd640 1 -- 172.21.15.151:0/202817389 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe248052470 con 0x7fe25013f650 2024-03-29T10:43:25.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.763+0000 7fe246ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.763+0000 7fe246ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:25.748 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:25.748 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:25.752 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.769+0000 7fe254b6d640 10 librados: watch_flush enter 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 10 librados: watch_flush exit 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 10 client.4472.objecter successfully canceled tick 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe23006dc00 legacy=0x7fe2300700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 10 monclient: shutdown 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:25.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 >> v1:172.21.15.151:6790/0 conn(0x7fe25013f650 legacy=0x7fe2501403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 shutdown_connections 2024-03-29T10:43:25.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 >> 172.21.15.151:0/202817389 conn(0x7fe250066a00 msgr2=0x7fe250137160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:25.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.770+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 shutdown_connections 2024-03-29T10:43:25.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.771+0000 7fe254b6d640 1 -- 172.21.15.151:0/202817389 wait complete. 2024-03-29T10:43:25.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:25.771+0000 7fe254b6d640 1 librados: shutdown 2024-03-29T10:43:25.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:32.581 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.736+0000 7f621713d640 10 monclient: get_monmap_and_config 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.736+0000 7f621713d640 10 monclient: build_initial_monmap 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.736+0000 7f621713d640 10 monclient: monmap: 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:32.737819+0000 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:32.737819+0000 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:32.720 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:32.721 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:32.721 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.737+0000 7f621713d640 1 Processor -- start 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.737+0000 7f621713d640 1 -- start start 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.737+0000 7f621713d640 10 monclient: init 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient(hunting): picked mon.noname-c con 0x7f62100b0770 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient(hunting): picked mon.noname-b con 0x7f621014a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient(hunting): picked mon.noname-a con 0x7f62100ac5a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62100aba70 con 0x7f62100ac5a0 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62100aecb0 con 0x7f62100b0770 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62100af150 con 0x7f621014a700 2024-03-29T10:43:32.722 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:32.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.738+0000 7f621713d640 10 monclient(hunting): authenticate will time out at 1343.908318s 2024-03-29T10:43:32.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f62156da640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f62100ac5a0 0x7f6210151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60042/0 (socket says 172.21.15.151:60042) 2024-03-29T10:43:32.723 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f62156da640 1 -- 172.21.15.151:0/238137731 learned_addr learned my addr 172.21.15.151:0/238137731 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601060691 0 0) 0x7f62100aba70 con 0x7f62100ac5a0 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc003520 con 0x7f62100ac5a0 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394310925 0 0) 0x7f62100af150 con 0x7f621014a700 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.724 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.739+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62100aba70 con 0x7f621014a700 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46260182 0 0) 0x7f62100aecb0 con 0x7f62100b0770 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62100af150 con 0x7f62100b0770 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339347725 0 0) 0x7f61fc003520 con 0x7f62100ac5a0 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 >> v1:172.21.15.151:6790/0 conn(0x7f621014a700 legacy=0x7f62100abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 >> v1:172.21.15.169:6789/0 conn(0x7f62100b0770 legacy=0x7f6210149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 monclient: found mon.noname-a 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:43:32.725 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62100ac9d0 con 0x7f62100ac5a0 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6200001940 con 0x7f62100ac5a0 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f620f7fe640 10 monclient: dump: 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.726 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.727 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.727 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.727 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f621713d640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:43:32.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.740+0000 7f621713d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6200001aa0 con 0x7f62100ac5a0 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 1 -- 172.21.15.151:0/238137731 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6200001ef0 con 0x7f62100ac5a0 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f620f7fe640 10 monclient: dump: 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.735 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f621713d640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.741+0000 7f621713d640 4 set_mon_vals no callback set 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 10 monclient: shutdown 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 1 -- 172.21.15.151:0/238137731 >> v1:172.21.15.151:6789/0 conn(0x7f62100ac5a0 legacy=0x7f6210151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 1 -- 172.21.15.151:0/238137731 shutdown_connections 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 1 -- 172.21.15.151:0/238137731 >> 172.21.15.151:0/238137731 conn(0x7f621006b660 msgr2=0x7f621006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 1 -- 172.21.15.151:0/238137731 shutdown_connections 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.742+0000 7f621713d640 1 -- 172.21.15.151:0/238137731 wait complete. 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.743+0000 7f621713d640 10 monclient: build_initial_monmap 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.743+0000 7f621713d640 10 monclient: monmap: 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:32.737 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 librados: starting msgr at 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 librados: starting objecter 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 Processor -- start 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 -- start start 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 librados: setting wanted keys 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 1 librados: calling monclient init 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.744+0000 7f621713d640 10 monclient: init 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient(hunting): picked mon.noname-a con 0x7f62100ac5a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient(hunting): picked mon.noname-c con 0x7f62100b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:32.738 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient(hunting): picked mon.noname-b con 0x7f621014a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6210145590 con 0x7f62100ac5a0 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6210128090 con 0x7f621014a700 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62101df3c0 con 0x7f62100b0770 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f621713d640 10 monclient(hunting): authenticate will time out at 1343.915244s 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f620ffff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f62100b0770 0x7f6210144770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:47142/0 (socket says 172.21.15.151:47142) 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.745+0000 7f620ffff640 1 -- 172.21.15.151:0/436892300 learned_addr learned my addr 172.21.15.151:0/436892300 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 client.?.objecter ms_handle_connect 0x7f62100ac5a0 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 client.?.objecter ms_handle_connect 0x7f62100b0770 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524293420 0 0) 0x7f62101df3c0 con 0x7f62100b0770 2024-03-29T10:43:32.739 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.746+0000 7f620d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f0004eb0 con 0x7f62100b0770 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 client.?.objecter ms_handle_connect 0x7f621014a700 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609073969 0 0) 0x7f6210145590 con 0x7f62100ac5a0 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62101df3c0 con 0x7f62100ac5a0 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847597849 0 0) 0x7f6210128090 con 0x7f621014a700 2024-03-29T10:43:32.740 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6210145590 con 0x7f621014a700 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520090394 0 0) 0x7f61f0004eb0 con 0x7f62100b0770 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6210128090 con 0x7f62100b0770 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f61f8001890 con 0x7f62100b0770 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.747+0000 7f620d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 838178492 0 0) 0x7f6210145590 con 0x7f621014a700 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f0004eb0 con 0x7f621014a700 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6200001c10 con 0x7f621014a700 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.743 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293988807 0 0) 0x7f62101df3c0 con 0x7f62100ac5a0 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6210145590 con 0x7f62100ac5a0 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6208001990 con 0x7f62100ac5a0 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2298289757 0 0) 0x7f61f0004eb0 con 0x7f621014a700 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 >> v1:172.21.15.151:6790/0 conn(0x7f62100b0770 legacy=0x7f6210144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 >> v1:172.21.15.151:6789/0 conn(0x7f62100ac5a0 legacy=0x7f62101db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 monclient: found mon.b 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62101457b0 con 0x7f621014a700 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f620d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f621713d640 5 monclient: authenticate success, global_id 4462 2024-03-29T10:43:32.745 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f621713d640 10 monclient: _renew_subs 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f621713d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f621713d640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62101dee10 con 0x7f621014a700 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.748+0000 7f621713d640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f621713d640 10 monclient: _renew_subs 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f621713d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f621713d640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6210070ef0 con 0x7f621014a700 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f621713d640 1 librados: init done 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6200001e90 con 0x7f621014a700 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.749+0000 7f620d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.750+0000 7f620effd640 4 set_mon_vals no callback set 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6200004fb0 con 0x7f621014a700 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f620d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f620d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f620d7fa640 10 monclient: dump: 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:32.746 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f6206ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f6206ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f6206ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.751+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f4005080 con 0x7f621014a700 2024-03-29T10:43:32.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.752+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f620001af80 con 0x7f621014a700 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.753+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f6200084d50 con 0x7f621014a700 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.753+0000 7f620d7fa640 10 client.4462.objecter ms_dispatch 0x7f6210000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.753+0000 7f6214ed9640 10 monclient: get_auth_request con 0x7f61f006e100 auth_method 0 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.753+0000 7f620d7fa640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.753+0000 7f620d7fa640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.754+0000 7f620d7fa640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.754+0000 7f620d7fa640 10 client.4462.objecter ms_handle_connect 0x7f61f006e100 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.761+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6200050630 con 0x7f621014a700 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.761+0000 7f620d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:32.761+0000 7f620d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:33.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.096+0000 7f6206ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:33.079 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.096+0000 7f6206ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:33.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.096+0000 7f6206ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:33.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.096+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61f4005330 con 0x7f621014a700 2024-03-29T10:43:33.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.097+0000 7f620d7fa640 1 -- 172.21.15.151:0/436892300 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f6200052560 con 0x7f621014a700 2024-03-29T10:43:33.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.097+0000 7f620d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:33.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.097+0000 7f620d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:33.081 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:33.081 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 10 librados: watch_flush enter 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 10 librados: watch_flush exit 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 10 client.4462.objecter successfully canceled tick 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 >> v1:172.21.15.151:6816/1549873000 conn(0x7f61f006e100 legacy=0x7f61f00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 10 monclient: shutdown 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 >> v1:172.21.15.169:6789/0 conn(0x7f621014a700 legacy=0x7f6210144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 shutdown_connections 2024-03-29T10:43:33.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.100+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 >> 172.21.15.151:0/436892300 conn(0x7f621006b660 msgr2=0x7f62100abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:33.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.101+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 shutdown_connections 2024-03-29T10:43:33.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.101+0000 7f6206ffd640 1 -- 172.21.15.151:0/436892300 wait complete. 2024-03-29T10:43:33.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:33.101+0000 7f6206ffd640 1 librados: shutdown 2024-03-29T10:43:33.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:39.904 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.055+0000 7f472914d640 10 monclient: get_monmap_and_config 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.055+0000 7f472914d640 10 monclient: build_initial_monmap 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.055+0000 7f472914d640 10 monclient: monmap: 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.039 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:40.056844+0000 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:40.056844+0000 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:40.040 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.056+0000 7f472914d640 1 Processor -- start 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.056+0000 7f472914d640 1 -- start start 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.056+0000 7f472914d640 10 monclient: init 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.057+0000 7f472914d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.057+0000 7f472914d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.057+0000 7f472914d640 10 monclient(hunting): picked mon.noname-c con 0x7f47240b0860 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.057+0000 7f472914d640 10 monclient(hunting): picked mon.noname-a con 0x7f472413ef30 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472359e640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f472406d9d0 0x7f472414a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:35530/0 (socket says 172.21.15.151:35530) 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472359e640 1 -- 172.21.15.151:0/3594299170 learned_addr learned my addr 172.21.15.151:0/3594299170 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 10 monclient(hunting): picked mon.noname-b con 0x7f472406d9d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240b0c40 con 0x7f472413ef30 2024-03-29T10:43:40.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240aed10 con 0x7f47240b0860 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240ab1b0 con 0x7f472406d9d0 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f472914d640 10 monclient(hunting): authenticate will time out at 1351.227867s 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508181326 0 0) 0x7f47240ab1b0 con 0x7f472406d9d0 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c004630 con 0x7f472406d9d0 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484907560 0 0) 0x7f470c004630 con 0x7f472406d9d0 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 >> v1:172.21.15.169:6789/0 conn(0x7f47240b0860 legacy=0x7f472413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.043 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.058+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 >> v1:172.21.15.151:6789/0 conn(0x7f472413ef30 legacy=0x7f47240af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 1 monclient: found mon.noname-b 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240ab490 con 0x7f472406d9d0 2024-03-29T10:43:40.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: _finish_auth 0 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f471c001a10 con 0x7f472406d9d0 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.059+0000 7f4721d9b640 10 monclient: dump: 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:40.047 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f472914d640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f472914d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f471c001ca0 con 0x7f472406d9d0 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 1 -- 172.21.15.151:0/3594299170 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f471c0020b0 con 0x7f472406d9d0 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f4721d9b640 10 monclient: dump: 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.048 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f472914d640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.061+0000 7f472914d640 4 set_mon_vals no callback set 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.062+0000 7f472914d640 10 monclient: shutdown 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.062+0000 7f472914d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.062+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 >> v1:172.21.15.151:6790/0 conn(0x7f472406d9d0 legacy=0x7f472414a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.062+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 shutdown_connections 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.062+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 >> 172.21.15.151:0/3594299170 conn(0x7f472406b660 msgr2=0x7f472406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.063+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 shutdown_connections 2024-03-29T10:43:40.049 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.063+0000 7f472914d640 1 -- 172.21.15.151:0/3594299170 wait complete. 2024-03-29T10:43:40.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.064+0000 7f472914d640 10 monclient: build_initial_monmap 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.064+0000 7f472914d640 10 monclient: monmap: 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 librados: starting msgr at 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 librados: starting objecter 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 Processor -- start 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 -- start start 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 librados: setting wanted keys 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 1 librados: calling monclient init 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.065+0000 7f472914d640 10 monclient: init 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.066+0000 7f472914d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.066+0000 7f472914d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.066+0000 7f472914d640 10 monclient(hunting): picked mon.noname-c con 0x7f472406d9d0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f4722d9d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f472406d9d0 0x7f4724139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:36414/0 (socket says 172.21.15.151:36414) 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f4722d9d640 1 -- 172.21.15.151:0/2461820585 learned_addr learned my addr 172.21.15.151:0/2461820585 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f472914d640 10 monclient(hunting): picked mon.noname-a con 0x7f47240b0860 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f470b7fe640 10 client.?.objecter ms_handle_connect 0x7f472406d9d0 2024-03-29T10:43:40.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f470b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.067+0000 7f472914d640 10 monclient(hunting): picked mon.noname-b con 0x7f472413ef30 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472413bf50 con 0x7f47240b0860 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472413a670 con 0x7f472413ef30 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472413a800 con 0x7f472406d9d0 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f472914d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f472914d640 10 monclient(hunting): authenticate will time out at 1351.238201s 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207591049 0 0) 0x7f472413a800 con 0x7f472406d9d0 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.068+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4710004eb0 con 0x7f472406d9d0 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 client.?.objecter ms_handle_connect 0x7f472413ef30 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:40.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969442030 0 0) 0x7f472413a670 con 0x7f472413ef30 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4710006230 con 0x7f472413ef30 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3668373900 0 0) 0x7f4710004eb0 con 0x7f472406d9d0 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f472413a670 con 0x7f472406d9d0 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899453090 0 0) 0x7f4710006230 con 0x7f472413ef30 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4710004eb0 con 0x7f472413ef30 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4718001990 con 0x7f472406d9d0 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): dump: 2024-03-29T10:43:40.061 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 client.?.objecter ms_handle_connect 0x7f47240b0860 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f471c001830 con 0x7f472413ef30 2024-03-29T10:43:40.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.069+0000 7f470b7fe640 10 monclient(hunting): dump: 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3234023301 0 0) 0x7f4710004eb0 con 0x7f472413ef30 2024-03-29T10:43:40.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 >> v1:172.21.15.151:6790/0 conn(0x7f472406d9d0 legacy=0x7f4724139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 >> v1:172.21.15.151:6789/0 conn(0x7f47240b0860 legacy=0x7f47241da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 1 monclient: found mon.b 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47241391c0 con 0x7f472413ef30 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f470b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 10 monclient: _renew_subs 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4724139470 con 0x7f472413ef30 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:40.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 10 monclient: _renew_subs 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f472411b1c0 con 0x7f472413ef30 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.070+0000 7f472914d640 1 librados: init done 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f471c004d00 con 0x7f472413ef30 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f471c005010 con 0x7f472413ef30 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f470b7fe640 10 monclient: dump: 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:40.065 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f472914d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f472914d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f472914d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.071+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472413c0e0 con 0x7f472413ef30 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.072+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f471c01afe0 con 0x7f472413ef30 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.072+0000 7f4720d99640 4 set_mon_vals no callback set 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f471c084db0 con 0x7f472413ef30 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 10 client.4465.objecter ms_dispatch 0x7f4724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:40.066 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f4722d9d640 10 monclient: get_auth_request con 0x7f471006e080 auth_method 0 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.073+0000 7f470b7fe640 10 client.4465.objecter ms_handle_connect 0x7f471006e080 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.080+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f471c0504c0 con 0x7f472413ef30 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.080+0000 7f470b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:40.067 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.080+0000 7f470b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:40.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.388+0000 7f472914d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.388+0000 7f472914d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.388+0000 7f472914d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:43:40.372 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.388+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4724075f10 con 0x7f472413ef30 2024-03-29T10:43:40.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.391+0000 7f470b7fe640 1 -- 172.21.15.151:0/2461820585 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f471c0525c0 con 0x7f472413ef30 2024-03-29T10:43:40.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.391+0000 7f470b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.391+0000 7f470b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:40.374 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:40.374 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 10 librados: watch_flush enter 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 10 librados: watch_flush exit 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 10 client.4465.objecter successfully canceled tick 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 >> v1:172.21.15.151:6816/1549873000 conn(0x7f471006e080 legacy=0x7f4710070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 10 monclient: shutdown 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:40.378 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.394+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 >> v1:172.21.15.169:6789/0 conn(0x7f472413ef30 legacy=0x7f47241db0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.395+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 shutdown_connections 2024-03-29T10:43:40.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.395+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 >> 172.21.15.151:0/2461820585 conn(0x7f472406b660 msgr2=0x7f47240ac840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:40.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.395+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 shutdown_connections 2024-03-29T10:43:40.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.395+0000 7f472914d640 1 -- 172.21.15.151:0/2461820585 wait complete. 2024-03-29T10:43:40.379 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:40.395+0000 7f472914d640 1 librados: shutdown 2024-03-29T10:43:40.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:47.202 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:47.347 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.364+0000 7fe457046640 10 monclient: get_monmap_and_config 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.364+0000 7fe457046640 10 monclient: build_initial_monmap 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.364+0000 7fe457046640 10 monclient: monmap: 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:47.365710+0000 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:47.365710+0000 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:47.348 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.351 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.365+0000 7fe457046640 1 Processor -- start 2024-03-29T10:43:47.351 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.365+0000 7fe457046640 1 -- start start 2024-03-29T10:43:47.351 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.365+0000 7fe457046640 10 monclient: init 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 10 monclient(hunting): picked mon.noname-b con 0x7fe4500b2800 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 10 monclient(hunting): picked mon.noname-a con 0x7fe4500a9c00 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 10 monclient(hunting): picked mon.noname-c con 0x7fe4500b38c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.366+0000 7fe457046640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe450141eb0 con 0x7fe4500a9c00 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.367+0000 7fe457046640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4500b22d0 con 0x7fe4500b38c0 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.367+0000 7fe457046640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe450142d30 con 0x7fe4500b2800 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.367+0000 7fe457046640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:47.352 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.367+0000 7fe457046640 10 monclient(hunting): authenticate will time out at 1358.536545s 2024-03-29T10:43:47.353 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.369+0000 7fe456044640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe4500b2800 0x7fe450142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:54540/0 (socket says 172.21.15.151:54540) 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.369+0000 7fe456044640 1 -- 172.21.15.151:0/2016111473 learned_addr learned my addr 172.21.15.151:0/2016111473 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224820818 0 0) 0x7fe450142d30 con 0x7fe4500b2800 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe43c003520 con 0x7fe4500b2800 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858654882 0 0) 0x7fe4500b22d0 con 0x7fe4500b38c0 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:43:47.354 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe450142d30 con 0x7fe4500b38c0 2024-03-29T10:43:47.355 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003942841 0 0) 0x7fe43c003520 con 0x7fe4500b2800 2024-03-29T10:43:47.355 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 >> v1:172.21.15.169:6789/0 conn(0x7fe4500b38c0 legacy=0x7fe45014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.355 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.370+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 >> v1:172.21.15.151:6789/0 conn(0x7fe4500a9c00 legacy=0x7fe45013bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:47.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 10 monclient: _finish_hunting 0 2024-03-29T10:43:47.357 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 1 monclient: found mon.noname-b 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4500aaf90 con 0x7fe4500b2800 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 10 monclient: _finish_auth 0 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 10 monclient: _check_auth_tickets 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.372+0000 7fe455042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe44c001690 con 0x7fe4500b2800 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: dump: 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:47.358 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe44c001870 con 0x7fe4500b2800 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 1 -- 172.21.15.151:0/2016111473 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe44c001c00 con 0x7fe4500b2800 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.373+0000 7fe455042640 10 monclient: dump: 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:47.359 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.374+0000 7fe457046640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.374+0000 7fe457046640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.374+0000 7fe457046640 4 set_mon_vals no callback set 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.375+0000 7fe457046640 10 monclient: shutdown 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.375+0000 7fe457046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:47.360 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.375+0000 7fe457046640 1 -- 172.21.15.151:0/2016111473 >> v1:172.21.15.151:6790/0 conn(0x7fe4500b2800 legacy=0x7fe450142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.375+0000 7fe457046640 1 -- 172.21.15.151:0/2016111473 shutdown_connections 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.375+0000 7fe457046640 1 -- 172.21.15.151:0/2016111473 >> 172.21.15.151:0/2016111473 conn(0x7fe450066a00 msgr2=0x7fe450066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.376+0000 7fe457046640 1 -- 172.21.15.151:0/2016111473 shutdown_connections 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.376+0000 7fe457046640 1 -- 172.21.15.151:0/2016111473 wait complete. 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 10 monclient: build_initial_monmap 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 10 monclient: monmap: 2024-03-29T10:43:47.365 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 librados: starting msgr at 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 librados: starting objecter 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 Processor -- start 2024-03-29T10:43:47.366 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 -- start start 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 librados: setting wanted keys 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 1 librados: calling monclient init 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.377+0000 7fe457046640 10 monclient: init 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.378+0000 7fe457046640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.378+0000 7fe457046640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.378+0000 7fe457046640 10 monclient(hunting): picked mon.noname-b con 0x7fe4500a9c00 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.378+0000 7fe457046640 10 monclient(hunting): picked mon.noname-a con 0x7fe4500b2800 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 10 monclient(hunting): picked mon.noname-c con 0x7fe4500b38c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45013eff0 con 0x7fe4500b2800 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4500ace50 con 0x7fe4500a9c00 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4500ad0c0 con 0x7fe4500b38c0 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe457046640 10 monclient(hunting): authenticate will time out at 1358.548589s 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe456845640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe4500b38c0 0x7fe4501e2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56326/0 (socket says 172.21.15.151:56326) 2024-03-29T10:43:47.367 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe456845640 1 -- 172.21.15.151:0/1772982891 learned_addr learned my addr 172.21.15.151:0/1772982891 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe446ffd640 10 client.?.objecter ms_handle_connect 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.379+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210224876 0 0) 0x7fe4500ad0c0 con 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe434004eb0 con 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707319211 0 0) 0x7fe434004eb0 con 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe434005300 con 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe448001bb0 con 0x7fe4500b38c0 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:47.368 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.380+0000 7fe446ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.381+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082850362 0 0) 0x7fe434005300 con 0x7fe4500b38c0 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.381+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 >> v1:172.21.15.169:6789/0 conn(0x7fe4500a9c00 legacy=0x7fe4501dab60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 >> v1:172.21.15.151:6789/0 conn(0x7fe4500b2800 legacy=0x7fe450137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:47.369 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 1 monclient: found mon.c 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45013f510 con 0x7fe4500b38c0 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe448003970 con 0x7fe4500b38c0 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe448001bb0 con 0x7fe4500b38c0 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe446ffd640 10 monclient: dump: 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:47.370 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:47.371 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.382+0000 7fe454841640 4 set_mon_vals no callback set 2024-03-29T10:43:47.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:43:47.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 10 monclient: _renew_subs 2024-03-29T10:43:47.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.373 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4500abf70 con 0x7fe4500b38c0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 10 monclient: _renew_subs 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4501db270 con 0x7fe4500b38c0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.383+0000 7fe457046640 1 librados: init done 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.385+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe448003ca0 con 0x7fe4500b38c0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.386+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe448054890 con 0x7fe4500b38c0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.386+0000 7fe446ffd640 10 client.4490.objecter ms_dispatch 0x7fe450000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.386+0000 7fe446ffd640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.386+0000 7fe446ffd640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.386+0000 7fe446ffd640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.387+0000 7fe457046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.387+0000 7fe457046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.374 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.387+0000 7fe457046640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.387+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45004f730 con 0x7fe4500b38c0 2024-03-29T10:43:47.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.389+0000 7fe456044640 10 monclient: get_auth_request con 0x7fe43406e020 auth_method 0 2024-03-29T10:43:47.375 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.389+0000 7fe446ffd640 10 client.4490.objecter ms_handle_connect 0x7fe43406e020 2024-03-29T10:43:47.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.398+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe448050b30 con 0x7fe4500b38c0 2024-03-29T10:43:47.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.399+0000 7fe446ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.382 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.399+0000 7fe446ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:47.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.730+0000 7fe457046640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.730+0000 7fe457046640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.730+0000 7fe457046640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:47.713 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.730+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4500670c0 con 0x7fe4500b38c0 2024-03-29T10:43:47.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.731+0000 7fe446ffd640 1 -- 172.21.15.151:0/1772982891 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe448050b30 con 0x7fe4500b38c0 2024-03-29T10:43:47.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.731+0000 7fe446ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.731+0000 7fe446ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:47.714 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:47.714 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:47.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.734+0000 7fe457046640 10 librados: watch_flush enter 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 10 librados: watch_flush exit 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 10 client.4490.objecter successfully canceled tick 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe43406e020 legacy=0x7fe4340704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 10 monclient: shutdown 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.735+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 >> v1:172.21.15.151:6790/0 conn(0x7fe4500b38c0 legacy=0x7fe4501e2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.736+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 shutdown_connections 2024-03-29T10:43:47.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.736+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 >> 172.21.15.151:0/1772982891 conn(0x7fe450066a00 msgr2=0x7fe45013bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:47.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.737+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 shutdown_connections 2024-03-29T10:43:47.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.737+0000 7fe457046640 1 -- 172.21.15.151:0/1772982891 wait complete. 2024-03-29T10:43:47.720 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:47.737+0000 7fe457046640 1 librados: shutdown 2024-03-29T10:43:47.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:54.545 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.714+0000 7ff1c1902640 10 monclient: get_monmap_and_config 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.714+0000 7ff1c1902640 10 monclient: build_initial_monmap 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.714+0000 7ff1c1902640 10 monclient: monmap: 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:43:54.715017+0000 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:43:54.715017+0000 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.697 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:54.698 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:43:54.698 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:43:54.698 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.715+0000 7ff1c1902640 1 Processor -- start 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.715+0000 7ff1c1902640 1 -- start start 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.715+0000 7ff1c1902640 10 monclient: init 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-b con 0x7ff1bc0b4330 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:54.700 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-c con 0x7ff1bc1470b0 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc1439b0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc06f2a0 con 0x7ff1bc1439b0 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc146b90 con 0x7ff1bc1470b0 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc0b2ca0 con 0x7ff1bc0b4330 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:54.701 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.716+0000 7ff1c1902640 10 monclient(hunting): authenticate will time out at 1365.885850s 2024-03-29T10:43:54.702 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.719+0000 7ff1ba7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff1bc1470b0 0x7ff1bc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:59059/0 (socket says 172.21.15.151:59059) 2024-03-29T10:43:54.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.719+0000 7ff1ba7fc640 1 -- 172.21.15.151:0/3003602386 learned_addr learned my addr 172.21.15.151:0/3003602386 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:54.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172043940 0 0) 0x7ff1bc146b90 con 0x7ff1bc1470b0 2024-03-29T10:43:54.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.720+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a4003520 con 0x7ff1bc1470b0 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372162953 0 0) 0x7ff1bc0b2ca0 con 0x7ff1bc0b4330 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1bc146b90 con 0x7ff1bc0b4330 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404637120 0 0) 0x7ff1a4003520 con 0x7ff1bc1470b0 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 >> v1:172.21.15.151:6790/0 conn(0x7ff1bc0b4330 legacy=0x7ff1bc0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 >> v1:172.21.15.151:6789/0 conn(0x7ff1bc1439b0 legacy=0x7ff1bc151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 monclient: found mon.noname-c 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc06e0c0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff1b4001a10 con 0x7ff1bc1470b0 2024-03-29T10:43:54.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1c1902640 5 monclient: authenticate success, global_id 4471 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.721+0000 7ff1b9ffb640 10 monclient: dump: 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.722+0000 7ff1c1902640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:54.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1b4001b70 con 0x7ff1bc1470b0 2024-03-29T10:43:54.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:54.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 1 -- 172.21.15.151:0/3003602386 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff1b4001f80 con 0x7ff1bc1470b0 2024-03-29T10:43:54.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.723+0000 7ff1b9ffb640 10 monclient: dump: 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.725+0000 7ff1c1902640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:54.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.725+0000 7ff1c1902640 4 set_mon_vals no callback set 2024-03-29T10:43:54.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.726+0000 7ff1c1902640 10 monclient: shutdown 2024-03-29T10:43:54.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.726+0000 7ff1c1902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:54.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.726+0000 7ff1c1902640 1 -- 172.21.15.151:0/3003602386 >> v1:172.21.15.169:6789/0 conn(0x7ff1bc1470b0 legacy=0x7ff1bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.727+0000 7ff1c1902640 1 -- 172.21.15.151:0/3003602386 shutdown_connections 2024-03-29T10:43:54.711 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.727+0000 7ff1c1902640 1 -- 172.21.15.151:0/3003602386 >> 172.21.15.151:0/3003602386 conn(0x7ff1bc066a00 msgr2=0x7ff1bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:54.712 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.729+0000 7ff1c1902640 1 -- 172.21.15.151:0/3003602386 shutdown_connections 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.729+0000 7ff1c1902640 1 -- 172.21.15.151:0/3003602386 wait complete. 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.730+0000 7ff1c1902640 10 monclient: build_initial_monmap 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.730+0000 7ff1c1902640 10 monclient: monmap: 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:43:54.714 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:43:54.715 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.730+0000 7ff1c1902640 1 librados: starting msgr at 2024-03-29T10:43:54.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.730+0000 7ff1c1902640 1 librados: starting objecter 2024-03-29T10:43:54.715 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.731+0000 7ff1c1902640 1 Processor -- start 2024-03-29T10:43:54.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.731+0000 7ff1c1902640 1 -- start start 2024-03-29T10:43:54.716 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.731+0000 7ff1c1902640 1 librados: setting wanted keys 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.731+0000 7ff1c1902640 1 librados: calling monclient init 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.731+0000 7ff1c1902640 10 monclient: init 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-b con 0x7ff1bc0b4330 addr v1:172.21.15.169:6789/0 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc1439b0 addr v1:172.21.15.151:6789/0 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient(hunting): picked mon.noname-c con 0x7ff1bc1470b0 addr v1:172.21.15.151:6790/0 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc1df7c0 con 0x7ff1bc1439b0 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc1df450 con 0x7ff1bc0b4330 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc141620 con 0x7ff1bc1470b0 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:54.717 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.732+0000 7ff1c1902640 10 monclient(hunting): authenticate will time out at 1365.901885s 2024-03-29T10:43:54.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.736+0000 7ff1bb7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7ff1bc1470b0 0x7ff1bc11f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:40168/0 (socket says 172.21.15.151:40168) 2024-03-29T10:43:54.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.736+0000 7ff1bb7fe640 1 -- 172.21.15.151:0/2379604985 learned_addr learned my addr 172.21.15.151:0/2379604985 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:43:54.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.736+0000 7ff1a3fff640 10 client.?.objecter ms_handle_connect 0x7ff1bc1470b0 2024-03-29T10:43:54.719 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.736+0000 7ff1a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 client.?.objecter ms_handle_connect 0x7ff1bc0b4330 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146454565 0 0) 0x7ff1bc141620 con 0x7ff1bc1470b0 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.726 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8004eb0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 client.?.objecter ms_handle_connect 0x7ff1bc1439b0 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520759940 0 0) 0x7ff1bc1df450 con 0x7ff1bc0b4330 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.739+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1bc141620 con 0x7ff1bc0b4330 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717480095 0 0) 0x7ff1a8004eb0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1bc1df450 con 0x7ff1bc1470b0 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff1b00018e0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.727 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): dump: 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906516763 0 0) 0x7ff1bc1df7c0 con 0x7ff1bc1439b0 2024-03-29T10:43:54.728 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a8004eb0 con 0x7ff1bc1439b0 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3179145617 0 0) 0x7ff1bc141620 con 0x7ff1bc0b4330 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.740+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1bc1df7c0 con 0x7ff1bc0b4330 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2501448694 0 0) 0x7ff1bc1df450 con 0x7ff1bc1470b0 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 >> v1:172.21.15.169:6789/0 conn(0x7ff1bc0b4330 legacy=0x7ff1bc1ded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 >> v1:172.21.15.151:6789/0 conn(0x7ff1bc1439b0 legacy=0x7ff1bc140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 1 monclient: found mon.c 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:54.729 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc1417b0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 10 monclient: _renew_subs 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc1d1860 con 0x7ff1bc1470b0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 10 monclient: _renew_subs 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc140d40 con 0x7ff1bc1470b0 2024-03-29T10:43:54.730 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.741+0000 7ff1c1902640 1 librados: init done 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.744+0000 7ff1c1902640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.744+0000 7ff1c1902640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.744+0000 7ff1c1902640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.744+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc1df5e0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1b00035f0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.731 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff1b00037a0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.745+0000 7ff1a3fff640 10 monclient: dump: 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:43:54.732 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.746+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7ff1b00039c0 con 0x7ff1bc1470b0 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.746+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7ff1b0084760 con 0x7ff1bc1470b0 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1a3fff640 10 client.4496.objecter ms_dispatch 0x7ff1bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1a3fff640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1a3fff640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1baffd640 10 monclient: get_auth_request con 0x7ff1a806ddf0 auth_method 0 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1a3fff640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.747+0000 7ff1b8ff9640 4 set_mon_vals no callback set 2024-03-29T10:43:54.733 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.748+0000 7ff1a3fff640 10 client.4496.objecter ms_handle_connect 0x7ff1a806ddf0 2024-03-29T10:43:54.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.753+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1b0050120 con 0x7ff1bc1470b0 2024-03-29T10:43:54.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.753+0000 7ff1a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.736 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:54.753+0000 7ff1a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:55.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.070+0000 7ff1c1902640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:55.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.070+0000 7ff1c1902640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:55.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.070+0000 7ff1c1902640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:43:55.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.070+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1bc075f10 con 0x7ff1bc1470b0 2024-03-29T10:43:55.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.071+0000 7ff1a3fff640 1 -- 172.21.15.151:0/2379604985 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7ff1b0052050 con 0x7ff1bc1470b0 2024-03-29T10:43:55.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.071+0000 7ff1a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:55.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.071+0000 7ff1a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:55.060 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:43:55.061 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 10 librados: watch_flush enter 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 10 librados: watch_flush exit 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 10 client.4496.objecter successfully canceled tick 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 >> v1:172.21.15.151:6816/1549873000 conn(0x7ff1a806ddf0 legacy=0x7ff1a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 10 monclient: shutdown 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:55.062 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.078+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 >> v1:172.21.15.151:6790/0 conn(0x7ff1bc1470b0 legacy=0x7ff1bc11f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:55.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.081+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 shutdown_connections 2024-03-29T10:43:55.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.081+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 >> 172.21.15.151:0/2379604985 conn(0x7ff1bc066a00 msgr2=0x7ff1bc142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:55.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.082+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 shutdown_connections 2024-03-29T10:43:55.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.082+0000 7ff1c1902640 1 -- 172.21.15.151:0/2379604985 wait complete. 2024-03-29T10:43:55.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:43:55.082+0000 7ff1c1902640 1 librados: shutdown 2024-03-29T10:43:55.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:01.892 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:02.038 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.054+0000 7ffa7d472640 10 monclient: get_monmap_and_config 2024-03-29T10:44:02.038 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.054+0000 7ffa7d472640 10 monclient: build_initial_monmap 2024-03-29T10:44:02.038 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.054+0000 7ffa7d472640 10 monclient: monmap: 2024-03-29T10:44:02.038 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:02.056522+0000 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:02.056522+0000 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:02.039 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.041 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.055+0000 7ffa7d472640 1 Processor -- start 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.055+0000 7ffa7d472640 1 -- start start 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.055+0000 7ffa7d472640 10 monclient: init 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-b con 0x7ffa70038a10 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-a con 0x7ffa70038df0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-c con 0x7ffa70030690 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700c54f0 con 0x7ffa70038df0 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa70036460 con 0x7ffa70030690 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa70039870 con 0x7ffa70038a10 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:02.042 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.056+0000 7ffa7d472640 10 monclient(hunting): authenticate will time out at 1373.227219s 2024-03-29T10:44:02.045 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.062+0000 7ffa777fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ffa70030690 0x7ffa700d31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:41435/0 (socket says 172.21.15.151:41435) 2024-03-29T10:44:02.046 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.062+0000 7ffa777fe640 1 -- 172.21.15.151:0/4195376413 learned_addr learned my addr 172.21.15.151:0/4195376413 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:02.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113265065 0 0) 0x7ffa70036460 con 0x7ffa70030690 2024-03-29T10:44:02.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:02.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:02.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:02.047 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:44:02.048 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.063+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60003520 con 0x7ffa70030690 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562508516 0 0) 0x7ffa60003520 con 0x7ffa70030690 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 >> v1:172.21.15.151:6790/0 conn(0x7ffa70038a10 legacy=0x7ffa70033550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 >> v1:172.21.15.151:6789/0 conn(0x7ffa70038df0 legacy=0x7ffa700cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 monclient: found mon.noname-c 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa700356c0 con 0x7ffa70030690 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:02.051 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ffa6c001690 con 0x7ffa70030690 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa75ffb640 10 monclient: dump: 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:02.052 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa7d472640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:44:02.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.067+0000 7ffa7d472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:02.053 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa6c001860 con 0x7ffa70030690 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 1 -- 172.21.15.151:0/4195376413 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ffa6c001c70 con 0x7ffa70030690 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa75ffb640 10 monclient: dump: 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:02.054 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa7d472640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.068+0000 7ffa7d472640 4 set_mon_vals no callback set 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.069+0000 7ffa7d472640 10 monclient: shutdown 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.069+0000 7ffa7d472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.069+0000 7ffa7d472640 1 -- 172.21.15.151:0/4195376413 >> v1:172.21.15.169:6789/0 conn(0x7ffa70030690 legacy=0x7ffa700d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.069+0000 7ffa7d472640 1 -- 172.21.15.151:0/4195376413 shutdown_connections 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.069+0000 7ffa7d472640 1 -- 172.21.15.151:0/4195376413 >> 172.21.15.151:0/4195376413 conn(0x7ffa70005c30 msgr2=0x7ffa70006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:02.055 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.071+0000 7ffa7d472640 1 -- 172.21.15.151:0/4195376413 shutdown_connections 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.072+0000 7ffa7d472640 1 -- 172.21.15.151:0/4195376413 wait complete. 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.073+0000 7ffa7d472640 10 monclient: build_initial_monmap 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.073+0000 7ffa7d472640 10 monclient: monmap: 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:02.058 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:02.059 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:02.059 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.073+0000 7ffa7d472640 1 librados: starting msgr at 2024-03-29T10:44:02.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.073+0000 7ffa7d472640 1 librados: starting objecter 2024-03-29T10:44:02.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.073+0000 7ffa7d472640 1 Processor -- start 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.076+0000 7ffa7d472640 1 -- start start 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.076+0000 7ffa7d472640 1 librados: setting wanted keys 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.076+0000 7ffa7d472640 1 librados: calling monclient init 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.076+0000 7ffa7d472640 10 monclient: init 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-b con 0x7ffa70030690 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-a con 0x7ffa70038a10 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7002e8e0 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:02.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700c73d0 con 0x7ffa70038a10 2024-03-29T10:44:02.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7000d920 con 0x7ffa70030690 2024-03-29T10:44:02.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700c7f20 con 0x7ffa7002e8e0 2024-03-29T10:44:02.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:02.065 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.077+0000 7ffa7d472640 10 monclient(hunting): authenticate will time out at 1373.248006s 2024-03-29T10:44:02.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa76ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ffa70030690 0x7ffa700c69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:41549/0 (socket says 172.21.15.151:41549) 2024-03-29T10:44:02.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa76ffd640 1 -- 172.21.15.151:0/2163572215 learned_addr learned my addr 172.21.15.151:0/2163572215 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:02.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 client.?.objecter ms_handle_connect 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415648582 0 0) 0x7ffa7000d920 con 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.083+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa64004eb0 con 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 client.?.objecter ms_handle_connect 0x7ffa70038a10 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2861469285 0 0) 0x7ffa64004eb0 con 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa7000d920 con 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ffa7804fdd0 con 0x7ffa70030690 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:02.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): dump: 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696627628 0 0) 0x7ffa700c73d0 con 0x7ffa70038a10 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:02.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa64004eb0 con 0x7ffa70038a10 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1519165836 0 0) 0x7ffa7000d920 con 0x7ffa70030690 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 >> v1:172.21.15.151:6790/0 conn(0x7ffa7002e8e0 legacy=0x7ffa7000c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 >> v1:172.21.15.151:6789/0 conn(0x7ffa70038a10 legacy=0x7ffa7002e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 monclient: found mon.b 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa700c8200 con 0x7ffa70030690 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:02.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.084+0000 7ffa5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:02.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa7804f400 con 0x7ffa70030690 2024-03-29T10:44:02.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:02.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa757fa640 4 set_mon_vals no callback set 2024-03-29T10:44:02.073 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 10 monclient: _renew_subs 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa700c84e0 con 0x7ffa70030690 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 10 monclient: _renew_subs 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa700c1d80 con 0x7ffa70030690 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa7d472640 1 librados: init done 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ffa78075710 con 0x7ffa70030690 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.085+0000 7ffa5f7fe640 10 monclient: dump: 2024-03-29T10:44:02.074 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.087+0000 7ffa7d472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.087+0000 7ffa7d472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.087+0000 7ffa7d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.087+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa70007bc0 con 0x7ffa70030690 2024-03-29T10:44:02.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.091+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7ffa78067bf0 con 0x7ffa70030690 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa767fc640 10 monclient: get_auth_request con 0x7ffa6406dc20 auth_method 0 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7ffa780ff5e0 con 0x7ffa70030690 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa5f7fe640 10 client.4480.objecter ms_dispatch 0x7ffa78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa5f7fe640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa5f7fe640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.092+0000 7ffa5f7fe640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T10:44:02.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.093+0000 7ffa5f7fe640 10 client.4480.objecter ms_handle_connect 0x7ffa6406dc20 2024-03-29T10:44:02.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.097+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa780caec0 con 0x7ffa70030690 2024-03-29T10:44:02.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.097+0000 7ffa5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:02.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.097+0000 7ffa5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:02.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.422+0000 7ffa7d472640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.422+0000 7ffa7d472640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.422+0000 7ffa7d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:02.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.422+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa70005150 con 0x7ffa70030690 2024-03-29T10:44:02.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.425+0000 7ffa5f7fe640 1 -- 172.21.15.151:0/2163572215 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7ffa780ccdf0 con 0x7ffa70030690 2024-03-29T10:44:02.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.425+0000 7ffa5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.425+0000 7ffa5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:02.411 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:02.411 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:02.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.429+0000 7ffa7d472640 10 librados: watch_flush enter 2024-03-29T10:44:02.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.429+0000 7ffa7d472640 10 librados: watch_flush exit 2024-03-29T10:44:02.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.429+0000 7ffa7d472640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T10:44:02.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.429+0000 7ffa7d472640 10 client.4480.objecter successfully canceled tick 2024-03-29T10:44:02.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.429+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 >> v1:172.21.15.151:6816/1549873000 conn(0x7ffa6406dc20 legacy=0x7ffa640700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.430+0000 7ffa7d472640 10 monclient: shutdown 2024-03-29T10:44:02.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.430+0000 7ffa7d472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:02.414 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.430+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 >> v1:172.21.15.169:6789/0 conn(0x7ffa70030690 legacy=0x7ffa700c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.432+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 shutdown_connections 2024-03-29T10:44:02.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.432+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 >> 172.21.15.151:0/2163572215 conn(0x7ffa70005c30 msgr2=0x7ffa700c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:02.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.433+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 shutdown_connections 2024-03-29T10:44:02.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.435+0000 7ffa7d472640 1 -- 172.21.15.151:0/2163572215 wait complete. 2024-03-29T10:44:02.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:02.436+0000 7ffa7d472640 1 librados: shutdown 2024-03-29T10:44:02.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:09.247 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.411+0000 7fc74116c640 10 monclient: get_monmap_and_config 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.411+0000 7fc74116c640 10 monclient: build_initial_monmap 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.411+0000 7fc74116c640 10 monclient: monmap: 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:09.413097+0000 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:09.413097+0000 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:09.395 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.396 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:09.396 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:09.396 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:09.396 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.397 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.413+0000 7fc74116c640 1 Processor -- start 2024-03-29T10:44:09.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.414+0000 7fc74116c640 1 -- start start 2024-03-29T10:44:09.398 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.414+0000 7fc74116c640 10 monclient: init 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-c con 0x7fc73c1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-a con 0x7fc73c13f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-b con 0x7fc73c0a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c0aaef0 con 0x7fc73c13f650 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c145bd0 con 0x7fc73c1457f0 2024-03-29T10:44:09.399 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c145eb0 con 0x7fc73c0a9d70 2024-03-29T10:44:09.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:09.400 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.415+0000 7fc74116c640 10 monclient(hunting): authenticate will time out at 1380.585815s 2024-03-29T10:44:09.402 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.418+0000 7fc73ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc73c1457f0 0x7fc73c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:43085/0 (socket says 172.21.15.151:43085) 2024-03-29T10:44:09.403 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.418+0000 7fc73ad9d640 1 -- 172.21.15.151:0/1065840623 learned_addr learned my addr 172.21.15.151:0/1065840623 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133217341 0 0) 0x7fc73c145bd0 con 0x7fc73c1457f0 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc724003520 con 0x7fc73c1457f0 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737084557 0 0) 0x7fc724003520 con 0x7fc73c1457f0 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.421+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 >> v1:172.21.15.151:6790/0 conn(0x7fc73c0a9d70 legacy=0x7fc73c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 >> v1:172.21.15.151:6789/0 conn(0x7fc73c13f650 legacy=0x7fc73c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 1 monclient: found mon.noname-c 2024-03-29T10:44:09.407 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c146340 con 0x7fc73c1457f0 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: _finish_auth 0 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc74116c640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc74116c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc72c001690 con 0x7fc73c1457f0 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc739d9b640 10 monclient: dump: 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:09.408 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.422+0000 7fc74116c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc72c001860 con 0x7fc73c1457f0 2024-03-29T10:44:09.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:09.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc74116c640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:09.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc74116c640 4 set_mon_vals no callback set 2024-03-29T10:44:09.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 1 -- 172.21.15.151:0/1065840623 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc72c001cb0 con 0x7fc73c1457f0 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.426+0000 7fc739d9b640 10 monclient: dump: 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:09.411 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 10 monclient: shutdown 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 1 -- 172.21.15.151:0/1065840623 >> v1:172.21.15.169:6789/0 conn(0x7fc73c1457f0 legacy=0x7fc73c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 1 -- 172.21.15.151:0/1065840623 shutdown_connections 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 1 -- 172.21.15.151:0/1065840623 >> 172.21.15.151:0/1065840623 conn(0x7fc73c066a00 msgr2=0x7fc73c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:09.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 1 -- 172.21.15.151:0/1065840623 shutdown_connections 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.427+0000 7fc74116c640 1 -- 172.21.15.151:0/1065840623 wait complete. 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.428+0000 7fc74116c640 10 monclient: build_initial_monmap 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.428+0000 7fc74116c640 10 monclient: monmap: 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:09.417 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.428+0000 7fc74116c640 1 librados: starting msgr at 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.428+0000 7fc74116c640 1 librados: starting objecter 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.429+0000 7fc74116c640 1 Processor -- start 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.429+0000 7fc74116c640 1 -- start start 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.429+0000 7fc74116c640 1 librados: setting wanted keys 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.429+0000 7fc74116c640 1 librados: calling monclient init 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.429+0000 7fc74116c640 10 monclient: init 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-a con 0x7fc73c0a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-c con 0x7fc73c13f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:09.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient(hunting): picked mon.noname-b con 0x7fc73c1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c070a50 con 0x7fc73c0a9d70 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c06fd10 con 0x7fc73c1457f0 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c06fea0 con 0x7fc73c13f650 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.430+0000 7fc74116c640 10 monclient(hunting): authenticate will time out at 1380.600760s 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.436+0000 7fc73a59c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fc73c13f650 0x7fc73c1db930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:46921/0 (socket says 172.21.15.151:46921) 2024-03-29T10:44:09.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.436+0000 7fc73a59c640 1 -- 172.21.15.151:0/3376796346 learned_addr learned my addr 172.21.15.151:0/3376796346 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:09.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 client.?.objecter ms_handle_connect 0x7fc73c13f650 2024-03-29T10:44:09.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:09.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929624857 0 0) 0x7fc73c06fea0 con 0x7fc73c13f650 2024-03-29T10:44:09.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:44:09.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.437+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc728004eb0 con 0x7fc73c13f650 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.439+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369911114 0 0) 0x7fc728004eb0 con 0x7fc73c13f650 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc73c06fea0 con 0x7fc73c13f650 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 client.?.objecter ms_handle_connect 0x7fc73c0a9d70 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc734001d50 con 0x7fc73c13f650 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:09.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): dump: 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652477759 0 0) 0x7fc73c070a50 con 0x7fc73c0a9d70 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc728004eb0 con 0x7fc73c0a9d70 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1349056465 0 0) 0x7fc73c06fea0 con 0x7fc73c13f650 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 >> v1:172.21.15.169:6789/0 conn(0x7fc73c1457f0 legacy=0x7fc73c1dc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 >> v1:172.21.15.151:6789/0 conn(0x7fc73c0a9d70 legacy=0x7fc73c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 monclient: found mon.c 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c070030 con 0x7fc73c13f650 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:09.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc734001eb0 con 0x7fc73c13f650 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.440+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fc734003c40 con 0x7fc73c13f650 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc7237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc7237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc7237fe640 10 monclient: dump: 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:09.428 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 10 monclient: _renew_subs 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c1420a0 con 0x7fc73c13f650 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 10 monclient: _renew_subs 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c11f9e0 con 0x7fc73c13f650 2024-03-29T10:44:09.429 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.441+0000 7fc74116c640 1 librados: init done 2024-03-29T10:44:09.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.443+0000 7fc738d99640 4 set_mon_vals no callback set 2024-03-29T10:44:09.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.443+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fc734003da0 con 0x7fc73c13f650 2024-03-29T10:44:09.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fc734084d80 con 0x7fc73c13f650 2024-03-29T10:44:09.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc7237fe640 10 client.4499.objecter ms_dispatch 0x7fc73c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:09.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc7237fe640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc7237fe640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc7237fe640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.444+0000 7fc73ad9d640 10 monclient: get_auth_request con 0x7fc72806dc10 auth_method 0 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.445+0000 7fc74116c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.445+0000 7fc74116c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.445+0000 7fc74116c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.445+0000 7fc7237fe640 10 client.4499.objecter ms_handle_connect 0x7fc72806dc10 2024-03-29T10:44:09.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.445+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c005080 con 0x7fc73c13f650 2024-03-29T10:44:09.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.453+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc734050490 con 0x7fc73c13f650 2024-03-29T10:44:09.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.453+0000 7fc7237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.453+0000 7fc7237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:09.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.780+0000 7fc74116c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.780+0000 7fc74116c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.780+0000 7fc74116c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:09.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.780+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc71c005330 con 0x7fc73c13f650 2024-03-29T10:44:09.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.783+0000 7fc7237fe640 1 -- 172.21.15.151:0/3376796346 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fc734052590 con 0x7fc73c13f650 2024-03-29T10:44:09.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.783+0000 7fc7237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.783+0000 7fc7237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:09.768 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:09.768 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 10 librados: watch_flush enter 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 10 librados: watch_flush exit 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 >> v1:172.21.15.151:6816/1549873000 conn(0x7fc72806dc10 legacy=0x7fc7280700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 10 monclient: shutdown 2024-03-29T10:44:09.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:09.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.787+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 >> v1:172.21.15.151:6790/0 conn(0x7fc73c13f650 legacy=0x7fc73c1db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.788+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 shutdown_connections 2024-03-29T10:44:09.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.788+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 >> 172.21.15.151:0/3376796346 conn(0x7fc73c066a00 msgr2=0x7fc73c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:09.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.788+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 shutdown_connections 2024-03-29T10:44:09.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.788+0000 7fc74116c640 1 -- 172.21.15.151:0/3376796346 wait complete. 2024-03-29T10:44:09.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:09.789+0000 7fc74116c640 1 librados: shutdown 2024-03-29T10:44:09.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:16.597 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:16.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.764+0000 7ff3c656d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:16.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.764+0000 7ff3c656d640 10 monclient: build_initial_monmap 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.764+0000 7ff3c656d640 10 monclient: monmap: 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:16.765090+0000 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:16.765090+0000 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.765+0000 7ff3c656d640 1 Processor -- start 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.765+0000 7ff3c656d640 1 -- start start 2024-03-29T10:44:16.749 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.765+0000 7ff3c656d640 10 monclient: init 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c00ae250 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c013f970 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c0142800 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00afe00 con 0x7ff3c013f970 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c0142c10 con 0x7ff3c0142800 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00afb20 con 0x7ff3c00ae250 2024-03-29T10:44:16.750 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:16.751 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.766+0000 7ff3c656d640 10 monclient(hunting): authenticate will time out at 1387.936100s 2024-03-29T10:44:16.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.769+0000 7ff3c4b0a640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff3c0142800 0x7ff3c014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:34931/0 (socket says 172.21.15.151:34931) 2024-03-29T10:44:16.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.769+0000 7ff3c4b0a640 1 -- 172.21.15.151:0/2767931741 learned_addr learned my addr 172.21.15.151:0/2767931741 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:16.753 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406456820 0 0) 0x7ff3c0142c10 con 0x7ff3c0142800 2024-03-29T10:44:16.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:16.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:16.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:16.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:44:16.754 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac003520 con 0x7ff3c0142800 2024-03-29T10:44:16.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.770+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438696550 0 0) 0x7ff3ac003520 con 0x7ff3c0142800 2024-03-29T10:44:16.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 >> v1:172.21.15.151:6790/0 conn(0x7ff3c00ae250 legacy=0x7ff3c013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 >> v1:172.21.15.151:6789/0 conn(0x7ff3c013f970 legacy=0x7ff3c01420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 monclient: found mon.noname-c 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00aab80 con 0x7ff3c0142800 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff3b8001a10 con 0x7ff3c0142800 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 10 monclient: dump: 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:16.757 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.771+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b8001ca0 con 0x7ff3c0142800 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3bf7fe640 1 -- 172.21.15.151:0/2767931741 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff3b8002030 con 0x7ff3c0142800 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:16.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3bf7fe640 10 monclient: dump: 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3c656d640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3c656d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.772+0000 7ff3c656d640 4 set_mon_vals no callback set 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.773+0000 7ff3c656d640 10 monclient: shutdown 2024-03-29T10:44:16.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.773+0000 7ff3c656d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:16.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.773+0000 7ff3c656d640 1 -- 172.21.15.151:0/2767931741 >> v1:172.21.15.169:6789/0 conn(0x7ff3c0142800 legacy=0x7ff3c014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.773+0000 7ff3c656d640 1 -- 172.21.15.151:0/2767931741 shutdown_connections 2024-03-29T10:44:16.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.773+0000 7ff3c656d640 1 -- 172.21.15.151:0/2767931741 >> 172.21.15.151:0/2767931741 conn(0x7ff3c00b35e0 msgr2=0x7ff3c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:16.760 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.778+0000 7ff3c656d640 1 -- 172.21.15.151:0/2767931741 shutdown_connections 2024-03-29T10:44:16.761 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.778+0000 7ff3c656d640 1 -- 172.21.15.151:0/2767931741 wait complete. 2024-03-29T10:44:16.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.782+0000 7ff3c656d640 10 monclient: build_initial_monmap 2024-03-29T10:44:16.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.782+0000 7ff3c656d640 10 monclient: monmap: 2024-03-29T10:44:16.765 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:16.766 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:16.767 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.782+0000 7ff3c656d640 1 librados: starting msgr at 2024-03-29T10:44:16.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.782+0000 7ff3c656d640 1 librados: starting objecter 2024-03-29T10:44:16.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.782+0000 7ff3c656d640 1 Processor -- start 2024-03-29T10:44:16.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 1 -- start start 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 1 librados: setting wanted keys 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 1 librados: calling monclient init 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 10 monclient: init 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.783+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c00ae250 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c013f970 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c0142800 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:16.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c0145640 con 0x7ff3c0142800 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c0146380 con 0x7ff3c013f970 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c0145120 con 0x7ff3c00ae250 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.784+0000 7ff3c656d640 10 monclient(hunting): authenticate will time out at 1387.953575s 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3c4b0a640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff3c0142800 0x7ff3c01e6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:54381/0 (socket says 172.21.15.151:54381) 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3c4b0a640 1 -- 172.21.15.151:0/1017370575 learned_addr learned my addr 172.21.15.151:0/1017370575 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 client.?.objecter ms_handle_connect 0x7ff3c0142800 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:16.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218218996 0 0) 0x7ff3c0145640 con 0x7ff3c0142800 2024-03-29T10:44:16.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:16.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:16.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:16.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:44:16.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.787+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0004eb0 con 0x7ff3c0142800 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 665858321 0 0) 0x7ff3a0004eb0 con 0x7ff3c0142800 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c0145640 con 0x7ff3c0142800 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff3b8001990 con 0x7ff3c0142800 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:16.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 10 monclient(hunting): dump: 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.789+0000 7ff3bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:16.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 client.?.objecter ms_handle_connect 0x7ff3c013f970 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988502645 0 0) 0x7ff3c0145640 con 0x7ff3c0142800 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 >> v1:172.21.15.151:6790/0 conn(0x7ff3c00ae250 legacy=0x7ff3c0144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 >> v1:172.21.15.169:6789/0 conn(0x7ff3c013f970 legacy=0x7ff3c01db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 1 monclient: found mon.a 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c01dbd50 con 0x7ff3c0142800 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:16.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3c656d640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3c656d640 10 monclient: _renew_subs 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3c656d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.791+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c01dbfa0 con 0x7ff3c0142800 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3c656d640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3c656d640 10 monclient: _renew_subs 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3c656d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c0144ba0 con 0x7ff3c0142800 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3c656d640 1 librados: init done 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b8004e10 con 0x7ff3c0142800 2024-03-29T10:44:16.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7ff3b8005120 con 0x7ff3c0142800 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3bd7fa640 10 monclient: dump: 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:16.784 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.792+0000 7ff3beffd640 4 set_mon_vals no callback set 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.793+0000 7ff3c656d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.793+0000 7ff3c656d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.793+0000 7ff3c656d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.793+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a4005080 con 0x7ff3c0142800 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7ff3b801b0f0 con 0x7ff3c0142800 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7ff3b8084ef0 con 0x7ff3c0142800 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 10 client.4464.objecter ms_dispatch 0x7ff3c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:16.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.794+0000 7ff3bd7fa640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T10:44:16.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.795+0000 7ff3bffff640 10 monclient: get_auth_request con 0x7ff3a006db80 auth_method 0 2024-03-29T10:44:16.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.795+0000 7ff3bd7fa640 10 client.4464.objecter ms_handle_connect 0x7ff3a006db80 2024-03-29T10:44:16.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.801+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3b80507d0 con 0x7ff3c0142800 2024-03-29T10:44:16.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.801+0000 7ff3bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:16.801+0000 7ff3bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:17.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.119+0000 7ff3c656d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:17.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.119+0000 7ff3c656d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:17.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.119+0000 7ff3c656d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:44:17.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.119+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3a4005330 con 0x7ff3c0142800 2024-03-29T10:44:17.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.122+0000 7ff3bd7fa640 1 -- 172.21.15.151:0/1017370575 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7ff3b8052700 con 0x7ff3c0142800 2024-03-29T10:44:17.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.122+0000 7ff3bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:17.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.122+0000 7ff3bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:17.106 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:17.106 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:17.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 10 librados: watch_flush enter 2024-03-29T10:44:17.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 10 librados: watch_flush exit 2024-03-29T10:44:17.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T10:44:17.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 10 client.4464.objecter successfully canceled tick 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 >> v1:172.21.15.151:6816/1549873000 conn(0x7ff3a006db80 legacy=0x7ff3a0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 10 monclient: shutdown 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 >> v1:172.21.15.151:6789/0 conn(0x7ff3c0142800 legacy=0x7ff3c01e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 shutdown_connections 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.126+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 >> 172.21.15.151:0/1017370575 conn(0x7ff3c00b35e0 msgr2=0x7ff3c00b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.127+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 shutdown_connections 2024-03-29T10:44:17.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.127+0000 7ff3c656d640 1 -- 172.21.15.151:0/1017370575 wait complete. 2024-03-29T10:44:17.111 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:17.128+0000 7ff3c656d640 1 librados: shutdown 2024-03-29T10:44:17.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:23.935 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.097+0000 7f4c8cfed640 10 monclient: get_monmap_and_config 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.097+0000 7f4c8cfed640 10 monclient: build_initial_monmap 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.097+0000 7f4c8cfed640 10 monclient: monmap: 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:24.098104+0000 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:24.098104+0000 2024-03-29T10:44:24.080 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.098+0000 7f4c8cfed640 1 Processor -- start 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.098+0000 7f4c8cfed640 1 -- start start 2024-03-29T10:44:24.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.098+0000 7f4c8cfed640 10 monclient: init 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-b con 0x7f4c880b0500 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-a con 0x7f4c880b08e0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-c con 0x7f4c880aeca0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c88140620 con 0x7f4c880b08e0 2024-03-29T10:44:24.083 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8813daf0 con 0x7f4c880aeca0 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880ae180 con 0x7f4c880b0500 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.099+0000 7f4c8cfed640 10 monclient(hunting): authenticate will time out at 1395.269457s 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c86d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4c880aeca0 0x7f4c8814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:49646/0 (socket says 172.21.15.151:49646) 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c86d9d640 1 -- 172.21.15.151:0/1318280701 learned_addr learned my addr 172.21.15.151:0/1318280701 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424113905 0 0) 0x7f4c88140620 con 0x7f4c880b08e0 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c70003520 con 0x7f4c880b08e0 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146014426 0 0) 0x7f4c8813daf0 con 0x7f4c880aeca0 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:44:24.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.100+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c88140620 con 0x7f4c880aeca0 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688132008 0 0) 0x7f4c880ae180 con 0x7f4c880b0500 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:44:24.087 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8813daf0 con 0x7f4c880b0500 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275321747 0 0) 0x7f4c70003520 con 0x7f4c880b08e0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 >> v1:172.21.15.151:6790/0 conn(0x7f4c880b0500 legacy=0x7f4c8813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 >> v1:172.21.15.169:6789/0 conn(0x7f4c880aeca0 legacy=0x7f4c8814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 monclient: found mon.noname-a 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880accf0 con 0x7f4c880b08e0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: _finish_auth 0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c7c001690 con 0x7f4c880b08e0 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:24.088 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: dump: 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8cfed640 5 monclient: authenticate success, global_id 4467 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8cfed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c7c001870 con 0x7f4c880b08e0 2024-03-29T10:44:24.089 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8cfed640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8cfed640 4 set_mon_vals no callback set 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 1 -- 172.21.15.151:0/1318280701 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c7c001690 con 0x7f4c880b08e0 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.101+0000 7f4c8559a640 10 monclient: dump: 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:24.090 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.102+0000 7f4c8cfed640 10 monclient: shutdown 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.102+0000 7f4c8cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.102+0000 7f4c8cfed640 1 -- 172.21.15.151:0/1318280701 >> v1:172.21.15.151:6789/0 conn(0x7f4c880b08e0 legacy=0x7f4c8813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.103+0000 7f4c8cfed640 1 -- 172.21.15.151:0/1318280701 shutdown_connections 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.103+0000 7f4c8cfed640 1 -- 172.21.15.151:0/1318280701 >> 172.21.15.151:0/1318280701 conn(0x7f4c8806b660 msgr2=0x7f4c8806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.103+0000 7f4c8cfed640 1 -- 172.21.15.151:0/1318280701 shutdown_connections 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.103+0000 7f4c8cfed640 1 -- 172.21.15.151:0/1318280701 wait complete. 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.104+0000 7f4c8cfed640 10 monclient: build_initial_monmap 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.104+0000 7f4c8cfed640 10 monclient: monmap: 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:24.091 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:24.092 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 librados: starting msgr at 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 librados: starting objecter 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 Processor -- start 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 -- start start 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 librados: setting wanted keys 2024-03-29T10:44:24.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 1 librados: calling monclient init 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.105+0000 7f4c8cfed640 10 monclient: init 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.106+0000 7f4c8cfed640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.106+0000 7f4c8cfed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.106+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-a con 0x7f4c880aeca0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.106+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-c con 0x7f4c880b0500 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.106+0000 7f4c8659c640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4c880aeca0 0x7f4c8813feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45500/0 (socket says 172.21.15.151:45500) 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8659c640 1 -- 172.21.15.151:0/668701972 learned_addr learned my addr 172.21.15.151:0/668701972 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter ms_handle_connect 0x7f4c880aeca0 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter ms_handle_connect 0x7f4c880b0500 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 10 monclient(hunting): picked mon.noname-b con 0x7f4c880700e0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8806e360 con 0x7f4c880aeca0 2024-03-29T10:44:24.101 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c881300f0 con 0x7f4c880700e0 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8806f9e0 con 0x7f4c880b0500 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c8cfed640 10 monclient(hunting): authenticate will time out at 1395.277105s 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter ms_handle_connect 0x7f4c880700e0 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.107+0000 7f4c76ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812983817 0 0) 0x7f4c8806f9e0 con 0x7f4c880b0500 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c68004eb0 con 0x7f4c880b0500 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834794304 0 0) 0x7f4c881300f0 con 0x7f4c880700e0 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.102 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8806f9e0 con 0x7f4c880700e0 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111948146 0 0) 0x7f4c8806e360 con 0x7f4c880aeca0 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c881300f0 con 0x7f4c880aeca0 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1504552377 0 0) 0x7f4c68004eb0 con 0x7f4c880b0500 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c8806e360 con 0x7f4c880b0500 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c7c001690 con 0x7f4c880b0500 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:24.103 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.108+0000 7f4c76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344608376 0 0) 0x7f4c8806e360 con 0x7f4c880b0500 2024-03-29T10:44:24.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 >> v1:172.21.15.169:6789/0 conn(0x7f4c880700e0 legacy=0x7f4c881e64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 >> v1:172.21.15.151:6789/0 conn(0x7f4c880aeca0 legacy=0x7f4c8813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 1 monclient: found mon.c 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8806fcc0 con 0x7f4c880b0500 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 10 monclient: _renew_subs 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8806dff0 con 0x7f4c880b0500 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:24.105 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 10 monclient: _renew_subs 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8806ec70 con 0x7f4c880b0500 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.109+0000 7f4c8cfed640 1 librados: init done 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c7c0011c0 con 0x7f4c880b0500 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4c7c003a40 con 0x7f4c880b0500 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c76ffd640 10 monclient: dump: 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:24.106 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.110+0000 7f4c84d99640 4 set_mon_vals no callback set 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.111+0000 7f4c8cfed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.111+0000 7f4c8cfed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.112+0000 7f4c8cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.112+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c005080 con 0x7f4c880b0500 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.112+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4c7c019a10 con 0x7f4c880b0500 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4c7c0838c0 con 0x7f4c880b0500 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 10 client.4508.objecter ms_dispatch 0x7f4c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:24.107 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c8659c640 10 monclient: get_auth_request con 0x7f4c6806ddf0 auth_method 0 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.113+0000 7f4c76ffd640 10 client.4508.objecter ms_handle_connect 0x7f4c6806ddf0 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.124+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c7c04f1a0 con 0x7f4c880b0500 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.124+0000 7f4c76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:24.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.124+0000 7f4c76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:24.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.498+0000 7f4c8cfed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.498+0000 7f4c8cfed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.498+0000 7f4c8cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:24.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.498+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6c005330 con 0x7f4c880b0500 2024-03-29T10:44:24.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.499+0000 7f4c76ffd640 1 -- 172.21.15.151:0/668701972 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4c7c0510d0 con 0x7f4c880b0500 2024-03-29T10:44:24.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.499+0000 7f4c76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.482 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:24.483 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:24.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.499+0000 7f4c76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 10 librados: watch_flush enter 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 10 librados: watch_flush exit 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 10 client.4508.objecter successfully canceled tick 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4c6806ddf0 legacy=0x7f4c680702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 10 monclient: shutdown 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:24.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 >> v1:172.21.15.151:6790/0 conn(0x7f4c880b0500 legacy=0x7f4c881350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 shutdown_connections 2024-03-29T10:44:24.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.503+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 >> 172.21.15.151:0/668701972 conn(0x7f4c8806b660 msgr2=0x7f4c880a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:24.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.504+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 shutdown_connections 2024-03-29T10:44:24.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.504+0000 7f4c8cfed640 1 -- 172.21.15.151:0/668701972 wait complete. 2024-03-29T10:44:24.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:24.504+0000 7f4c8cfed640 1 librados: shutdown 2024-03-29T10:44:24.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:31.309 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:31.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.483+0000 7f82b3d15640 10 monclient: get_monmap_and_config 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.483+0000 7f82b3d15640 10 monclient: build_initial_monmap 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.483+0000 7f82b3d15640 10 monclient: monmap: 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:31.484593+0000 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:31.484593+0000 2024-03-29T10:44:31.468 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.469 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.484+0000 7f82b3d15640 1 Processor -- start 2024-03-29T10:44:31.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.485+0000 7f82b3d15640 1 -- start start 2024-03-29T10:44:31.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.485+0000 7f82b3d15640 10 monclient: init 2024-03-29T10:44:31.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.486+0000 7f82b3d15640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:31.470 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.486+0000 7f82b3d15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:31.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.486+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-c con 0x7f82ac0a9ac0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:31.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-a con 0x7f82ac13ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:31.473 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-b con 0x7f82ac0af900 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0afd10 con 0x7f82ac13ee90 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0af6f0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0aac80 con 0x7f82ac0af900 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.487+0000 7f82b3d15640 10 monclient(hunting): authenticate will time out at 1402.656817s 2024-03-29T10:44:31.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.488+0000 7f82b22b2640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f82ac0af900 0x7f82ac149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:53386/0 (socket says 172.21.15.151:53386) 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b22b2640 1 -- 172.21.15.151:0/2419657373 learned_addr learned my addr 172.21.15.151:0/2419657373 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196923186 0 0) 0x7f82ac0aac80 con 0x7f82ac0af900 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 10 monclient(hunting): _negotiate 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:44:31.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.489+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298003520 con 0x7f82ac0af900 2024-03-29T10:44:31.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.490+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1521548305 0 0) 0x7f8298003520 con 0x7f82ac0af900 2024-03-29T10:44:31.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.490+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 >> v1:172.21.15.169:6789/0 conn(0x7f82ac0a9ac0 legacy=0x7f82ac0adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:44:31.478 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.490+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 >> v1:172.21.15.151:6789/0 conn(0x7f82ac13ee90 legacy=0x7f82ac13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: _finish_hunting 0 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 1 monclient: found mon.noname-b 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0aaf60 con 0x7f82ac0af900 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: _finish_auth 0 2024-03-29T10:44:31.481 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: _check_auth_tickets 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f82a0001690 con 0x7f82ac0af900 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b0aaf640 10 monclient: dump: 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.482 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b3d15640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:44:31.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.495+0000 7f82b3d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:31.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a0001860 con 0x7f82ac0af900 2024-03-29T10:44:31.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 1 -- 172.21.15.151:0/2419657373 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f82a0001cb0 con 0x7f82ac0af900 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b0aaf640 10 monclient: dump: 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.485 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b3d15640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.499+0000 7f82b3d15640 4 set_mon_vals no callback set 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.501+0000 7f82b3d15640 10 monclient: shutdown 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.501+0000 7f82b3d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.501+0000 7f82b3d15640 1 -- 172.21.15.151:0/2419657373 >> v1:172.21.15.151:6790/0 conn(0x7f82ac0af900 legacy=0x7f82ac149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.501+0000 7f82b3d15640 1 -- 172.21.15.151:0/2419657373 shutdown_connections 2024-03-29T10:44:31.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.501+0000 7f82b3d15640 1 -- 172.21.15.151:0/2419657373 >> 172.21.15.151:0/2419657373 conn(0x7f82ac0b35e0 msgr2=0x7f82ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:31.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.502+0000 7f82b3d15640 1 -- 172.21.15.151:0/2419657373 shutdown_connections 2024-03-29T10:44:31.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.502+0000 7f82b3d15640 1 -- 172.21.15.151:0/2419657373 wait complete. 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.503+0000 7f82b3d15640 10 monclient: build_initial_monmap 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.503+0000 7f82b3d15640 10 monclient: monmap: 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:31.488 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:31.489 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 librados: starting msgr at 2024-03-29T10:44:31.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 librados: starting objecter 2024-03-29T10:44:31.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 Processor -- start 2024-03-29T10:44:31.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 -- start start 2024-03-29T10:44:31.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 librados: setting wanted keys 2024-03-29T10:44:31.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 1 librados: calling monclient init 2024-03-29T10:44:31.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.504+0000 7f82b3d15640 10 monclient: init 2024-03-29T10:44:31.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.505+0000 7f82b3d15640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:31.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.505+0000 7f82b3d15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:31.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.505+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-b con 0x7f82ac0a9ac0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:31.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.505+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-c con 0x7f82ac0af900 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:31.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 10 monclient(hunting): picked mon.noname-a con 0x7f82ac13ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:31.495 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac070880 con 0x7f82ac13ee90 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac06f630 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac1421f0 con 0x7f82ac0af900 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b3d15640 10 monclient(hunting): authenticate will time out at 1402.678831s 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b12b0640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f82ac0af900 0x7f82ac1398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:53862/0 (socket says 172.21.15.151:53862) 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f82b12b0640 1 -- 172.21.15.151:0/845294157 learned_addr learned my addr 172.21.15.151:0/845294157 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f829e7fc640 10 client.?.objecter ms_handle_connect 0x7f82ac0a9ac0 2024-03-29T10:44:31.496 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.509+0000 7f829e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548172570 0 0) 0x7f82ac06f630 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.510+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8290004eb0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 client.?.objecter ms_handle_connect 0x7f82ac0af900 2024-03-29T10:44:31.497 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321511647 0 0) 0x7f8290004eb0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8290005300 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f82a4001b10 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:31.498 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.499 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.511+0000 7f829e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3964933506 0 0) 0x7f8290005300 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 >> v1:172.21.15.151:6790/0 conn(0x7f82ac0af900 legacy=0x7f82ac1398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 >> v1:172.21.15.151:6789/0 conn(0x7f82ac13ee90 legacy=0x7f82ac1d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 monclient: found mon.b 2024-03-29T10:44:31.500 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac1424b0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a40020a0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f82a4003b10 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.501 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.512+0000 7f829e7fc640 10 monclient: dump: 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:31.502 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 10 monclient: _renew_subs 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac13a000 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 10 monclient: _renew_subs 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:31.503 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac06e980 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.513+0000 7f82b3d15640 1 librados: init done 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.516+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f82a4019b60 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829f7fe640 4 set_mon_vals no callback set 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f82b3d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f82a40537a0 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829e7fc640 10 client.4498.objecter ms_dispatch 0x7f82ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829e7fc640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829e7fc640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.517+0000 7f829e7fc640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.518+0000 7f82b3d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.518+0000 7f82b3d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.518+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac075f10 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.521+0000 7f82b12b0640 10 monclient: get_auth_request con 0x7f8290072430 auth_method 0 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.521+0000 7f829e7fc640 10 client.4498.objecter ms_handle_connect 0x7f8290072430 2024-03-29T10:44:31.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.526+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82a404fa40 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.526+0000 7f829e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.509 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.526+0000 7f829e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:31.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.873+0000 7f82b3d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.873+0000 7f82b3d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.873+0000 7f82b3d15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:31.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.873+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82ac0ae640 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.874+0000 7f829e7fc640 1 -- 172.21.15.151:0/845294157 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f82a404fa40 con 0x7f82ac0a9ac0 2024-03-29T10:44:31.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.874+0000 7f829e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.874+0000 7f829e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:31.858 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:31.858 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:31.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 10 librados: watch_flush enter 2024-03-29T10:44:31.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 10 librados: watch_flush exit 2024-03-29T10:44:31.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T10:44:31.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 10 client.4498.objecter successfully canceled tick 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 >> v1:172.21.15.151:6816/1549873000 conn(0x7f8290072430 legacy=0x7f82900748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 10 monclient: shutdown 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 >> v1:172.21.15.169:6789/0 conn(0x7f82ac0a9ac0 legacy=0x7f82ac06ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 shutdown_connections 2024-03-29T10:44:31.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.878+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 >> 172.21.15.151:0/845294157 conn(0x7f82ac0b35e0 msgr2=0x7f82ac13b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:31.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.879+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 shutdown_connections 2024-03-29T10:44:31.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.879+0000 7f82b3d15640 1 -- 172.21.15.151:0/845294157 wait complete. 2024-03-29T10:44:31.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:31.879+0000 7f82b3d15640 1 librados: shutdown 2024-03-29T10:44:31.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:38.688 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.862+0000 7fe9f5674640 10 monclient: get_monmap_and_config 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.862+0000 7fe9f5674640 10 monclient: build_initial_monmap 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.862+0000 7fe9f5674640 10 monclient: monmap: 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:38.863015+0000 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:38.863015+0000 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:38.845 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:38.846 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:38.846 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.863+0000 7fe9f5674640 1 Processor -- start 2024-03-29T10:44:38.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.863+0000 7fe9f5674640 1 -- start start 2024-03-29T10:44:38.846 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.863+0000 7fe9f5674640 10 monclient: init 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e8038c60 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e80324b0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e80d11e0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9effff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe9e8038c60 0x7fe9e80c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:50818/0 (socket says 172.21.15.151:50818) 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80caf80 con 0x7fe9e80324b0 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9effff640 1 -- 172.21.15.151:0/831782408 learned_addr learned my addr 172.21.15.151:0/831782408 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:38.847 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e8039850 con 0x7fe9e80d11e0 2024-03-29T10:44:38.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80c78d0 con 0x7fe9e8038c60 2024-03-29T10:44:38.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:38.848 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9f5674640 10 monclient(hunting): authenticate will time out at 1410.034118s 2024-03-29T10:44:38.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078922685 0 0) 0x7fe9e8039850 con 0x7fe9e80d11e0 2024-03-29T10:44:38.850 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.864+0000 7fe9eeffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003520 con 0x7fe9e80d11e0 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92944280 0 0) 0x7fe9e80caf80 con 0x7fe9e80324b0 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e8039850 con 0x7fe9e80324b0 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283457597 0 0) 0x7fe9e80c78d0 con 0x7fe9e8038c60 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.851 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e80caf80 con 0x7fe9e8038c60 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.865+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487284964 0 0) 0x7fe9e8039850 con 0x7fe9e80324b0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 >> v1:172.21.15.151:6790/0 conn(0x7fe9e8038c60 legacy=0x7fe9e80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 >> v1:172.21.15.169:6789/0 conn(0x7fe9e80d11e0 legacy=0x7fe9e80d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80c7a60 con 0x7fe9e80324b0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe9dc001690 con 0x7fe9e80324b0 2024-03-29T10:44:38.852 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: dump: 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9f5674640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9f5674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:38.853 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9dc0018b0 con 0x7fe9e80324b0 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9f5674640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9f5674640 4 set_mon_vals no callback set 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 1 -- 172.21.15.151:0/831782408 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe9dc001c40 con 0x7fe9e80324b0 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.866+0000 7fe9eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9eeffd640 10 monclient: dump: 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 10 monclient: shutdown 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 >> v1:172.21.15.151:6789/0 conn(0x7fe9e80324b0 legacy=0x7fe9e80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 shutdown_connections 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 >> 172.21.15.151:0/831782408 conn(0x7fe9e8005c20 msgr2=0x7fe9e8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 shutdown_connections 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.867+0000 7fe9f5674640 1 -- 172.21.15.151:0/831782408 wait complete. 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.868+0000 7fe9f5674640 10 monclient: build_initial_monmap 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.868+0000 7fe9f5674640 10 monclient: monmap: 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.860 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 librados: starting msgr at 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 librados: starting objecter 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 Processor -- start 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 -- start start 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 librados: setting wanted keys 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 1 librados: calling monclient init 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.869+0000 7fe9f5674640 10 monclient: init 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.870+0000 7fe9f5674640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.870+0000 7fe9f5674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.870+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e80324b0 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.870+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e8038c60 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.870+0000 7fe9f5674640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e80d11e0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9f5674640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e8036730 con 0x7fe9e80d11e0 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9f5674640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e8034d70 con 0x7fe9e8038c60 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9f5674640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80bb8f0 con 0x7fe9e80324b0 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9f5674640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9f5674640 10 monclient(hunting): authenticate will time out at 1410.040636s 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9effff640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe9e80324b0 0x7fe9e8034660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:51076/0 (socket says 172.21.15.151:51076) 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9effff640 1 -- 172.21.15.151:0/3992963773 learned_addr learned my addr 172.21.15.151:0/3992963773 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 client.?.objecter ms_handle_connect 0x7fe9e80324b0 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 client.?.objecter ms_handle_connect 0x7fe9e8038c60 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496014677 0 0) 0x7fe9e8034d70 con 0x7fe9e8038c60 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d4004eb0 con 0x7fe9e8038c60 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 508576414 0 0) 0x7fe9d4004eb0 con 0x7fe9e8038c60 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.871+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e8034d70 con 0x7fe9e8038c60 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe9dc001690 con 0x7fe9e8038c60 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): dump: 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452912042 0 0) 0x7fe9e80bb8f0 con 0x7fe9e80324b0 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.864 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d4004eb0 con 0x7fe9e80324b0 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 client.?.objecter ms_handle_connect 0x7fe9e80d11e0 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717927016 0 0) 0x7fe9e8034d70 con 0x7fe9e8038c60 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 >> v1:172.21.15.151:6790/0 conn(0x7fe9e80324b0 legacy=0x7fe9e8034660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 >> v1:172.21.15.151:6789/0 conn(0x7fe9e80d11e0 legacy=0x7fe9e80bbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 monclient: found mon.b 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80bba80 con 0x7fe9e8038c60 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:38.865 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 5 monclient: authenticate success, global_id 4504 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 10 monclient: _renew_subs 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e8035030 con 0x7fe9e8038c60 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 10 monclient: _renew_subs 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80bc620 con 0x7fe9e8038c60 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.872+0000 7fe9f5674640 1 librados: init done 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9dc003920 con 0x7fe9e8038c60 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:38.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe9dc001250 con 0x7fe9e8038c60 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ecff9640 10 monclient: dump: 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:38.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.873+0000 7fe9ee7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.874+0000 7fe9f5674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.874+0000 7fe9f5674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.874+0000 7fe9f5674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d0005080 con 0x7fe9e8038c60 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe9dc003d30 con 0x7fe9e8038c60 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe9dc083a70 con 0x7fe9e8038c60 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 10 client.4504.objecter ms_dispatch 0x7fe9f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9effff640 10 monclient: get_auth_request con 0x7fe9d406dd00 auth_method 0 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.875+0000 7fe9ecff9640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T10:44:38.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.876+0000 7fe9ecff9640 10 client.4504.objecter ms_handle_connect 0x7fe9d406dd00 2024-03-29T10:44:38.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.885+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9dc04f3b0 con 0x7fe9e8038c60 2024-03-29T10:44:38.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.885+0000 7fe9ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:38.885+0000 7fe9ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:39.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.250+0000 7fe9f5674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:39.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.250+0000 7fe9f5674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:39.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.250+0000 7fe9f5674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:44:39.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.250+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9d0005330 con 0x7fe9e8038c60 2024-03-29T10:44:39.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.253+0000 7fe9ecff9640 1 -- 172.21.15.151:0/3992963773 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe9dc083120 con 0x7fe9e8038c60 2024-03-29T10:44:39.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.253+0000 7fe9ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:39.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.253+0000 7fe9ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:39.236 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:39.236 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:39.239 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 10 librados: watch_flush enter 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 10 librados: watch_flush exit 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 10 client.4504.objecter successfully canceled tick 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe9d406dd00 legacy=0x7fe9d40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 10 monclient: shutdown 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:39.240 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 >> v1:172.21.15.169:6789/0 conn(0x7fe9e8038c60 legacy=0x7fe9e80bb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:39.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.257+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 shutdown_connections 2024-03-29T10:44:39.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.258+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 >> 172.21.15.151:0/3992963773 conn(0x7fe9e8005c20 msgr2=0x7fe9e80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:39.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.258+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 shutdown_connections 2024-03-29T10:44:39.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.258+0000 7fe9f5674640 1 -- 172.21.15.151:0/3992963773 wait complete. 2024-03-29T10:44:39.241 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:39.258+0000 7fe9f5674640 1 librados: shutdown 2024-03-29T10:44:39.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:46.065 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 10 monclient: build_initial_monmap 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 10 monclient: monmap: 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:46.218059+0000 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:46.218059+0000 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:46.200 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 1 Processor -- start 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 1 -- start start 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.217+0000 7fa2f902e640 10 monclient: init 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f41457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:46.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f413f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f40a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40aaef0 con 0x7fa2f41457f0 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.218+0000 7fa2f902e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f4145bd0 con 0x7fa2f413f650 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f359e640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fa2f40a9d70 0x7fa2f414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:33380/0 (socket says 172.21.15.151:33380) 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f359e640 1 -- 172.21.15.151:0/2976101488 learned_addr learned my addr 172.21.15.151:0/2976101488 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f902e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f4145eb0 con 0x7fa2f40a9d70 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f902e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f902e640 10 monclient(hunting): authenticate will time out at 1417.388656s 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509848238 0 0) 0x7fa2f4145eb0 con 0x7fa2f40a9d70 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 10 monclient(hunting): _negotiate 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:44:46.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.219+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e0003520 con 0x7fa2f40a9d70 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095075823 0 0) 0x7fa2f4145bd0 con 0x7fa2f413f650 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 10 monclient(hunting): _negotiate 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4145eb0 con 0x7fa2f413f650 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061369647 0 0) 0x7fa2f4145eb0 con 0x7fa2f413f650 2024-03-29T10:44:46.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 >> v1:172.21.15.151:6790/0 conn(0x7fa2f40a9d70 legacy=0x7fa2f414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.220+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 >> v1:172.21.15.151:6789/0 conn(0x7fa2f41457f0 legacy=0x7fa2f413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: _finish_hunting 0 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 1 monclient: found mon.noname-c 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4146340 con 0x7fa2f413f650 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: _finish_auth 0 2024-03-29T10:44:46.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: _check_auth_tickets 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2dc001670 con 0x7fa2f413f650 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f259c640 10 monclient: dump: 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:46.206 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:46.207 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f902e640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:44:46.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.222+0000 7fa2f902e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:46.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2dc0017d0 con 0x7fa2f413f650 2024-03-29T10:44:46.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 1 -- 172.21.15.151:0/2976101488 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2dc001c20 con 0x7fa2f413f650 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f259c640 10 monclient: dump: 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f902e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:46.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.225+0000 7fa2f902e640 4 set_mon_vals no callback set 2024-03-29T10:44:46.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.226+0000 7fa2f902e640 10 monclient: shutdown 2024-03-29T10:44:46.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.226+0000 7fa2f902e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:46.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.226+0000 7fa2f902e640 1 -- 172.21.15.151:0/2976101488 >> v1:172.21.15.169:6789/0 conn(0x7fa2f413f650 legacy=0x7fa2f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.226+0000 7fa2f902e640 1 -- 172.21.15.151:0/2976101488 shutdown_connections 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.226+0000 7fa2f902e640 1 -- 172.21.15.151:0/2976101488 >> 172.21.15.151:0/2976101488 conn(0x7fa2f4066a00 msgr2=0x7fa2f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.227+0000 7fa2f902e640 1 -- 172.21.15.151:0/2976101488 shutdown_connections 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.227+0000 7fa2f902e640 1 -- 172.21.15.151:0/2976101488 wait complete. 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.228+0000 7fa2f902e640 10 monclient: build_initial_monmap 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.228+0000 7fa2f902e640 10 monclient: monmap: 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:46.211 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:46.212 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.229+0000 7fa2f902e640 1 librados: starting msgr at 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.229+0000 7fa2f902e640 1 librados: starting objecter 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.229+0000 7fa2f902e640 1 Processor -- start 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.230+0000 7fa2f902e640 1 -- start start 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.230+0000 7fa2f902e640 1 librados: setting wanted keys 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.230+0000 7fa2f902e640 1 librados: calling monclient init 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.230+0000 7fa2f902e640 10 monclient: init 2024-03-29T10:44:46.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:46.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:46.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f40a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:46.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f413f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:46.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f41457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40acde0 con 0x7fa2f40a9d70 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40ad020 con 0x7fa2f41457f0 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f41d71d0 con 0x7fa2f413f650 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.231+0000 7fa2f902e640 10 monclient(hunting): authenticate will time out at 1417.401076s 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2f2d9d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fa2f413f650 0x7fa2f41d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:34954/0 (socket says 172.21.15.151:34954) 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2f2d9d640 1 -- 172.21.15.151:0/3035896074 learned_addr learned my addr 172.21.15.151:0/3035896074 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 client.?.objecter ms_handle_connect 0x7fa2f413f650 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239385429 0 0) 0x7fa2f41d71d0 con 0x7fa2f413f650 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:44:46.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.232+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d4004eb0 con 0x7fa2f413f650 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99639268 0 0) 0x7fa2d4004eb0 con 0x7fa2f413f650 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f41d71d0 con 0x7fa2f413f650 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2dc001630 con 0x7fa2f413f650 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:46.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 10 monclient(hunting): dump: 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.233+0000 7fa2ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 10 client.?.objecter ms_handle_connect 0x7fa2f41457f0 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:46.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4191368491 0 0) 0x7fa2f41d71d0 con 0x7fa2f413f650 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 >> v1:172.21.15.169:6789/0 conn(0x7fa2f41457f0 legacy=0x7fa2f41e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 >> v1:172.21.15.151:6789/0 conn(0x7fa2f40a9d70 legacy=0x7fa2f40ab870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 1 monclient: found mon.c 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4138a90 con 0x7fa2f413f650 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.234+0000 7fa2ebfff640 10 monclient: _finish_auth 0 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2dc003910 con 0x7fa2f413f650 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fa2dc003ce0 con 0x7fa2f413f650 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:46.223 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2ebfff640 10 monclient: dump: 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 10 monclient: _renew_subs 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f159a640 4 set_mon_vals no callback set 2024-03-29T10:44:46.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f41d7de0 con 0x7fa2f413f650 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 10 monclient: _renew_subs 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f413b610 con 0x7fa2f413f650 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.235+0000 7fa2f902e640 1 librados: init done 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.236+0000 7fa2f902e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.236+0000 7fa2f902e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.236+0000 7fa2f902e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.236+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f4138c80 con 0x7fa2f413f650 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.237+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fa2dc003e40 con 0x7fa2f413f650 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fa2dc084c80 con 0x7fa2f413f650 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 10 client.4526.objecter ms_dispatch 0x7fa2f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:46.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:46.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ead9d640 10 monclient: get_auth_request con 0x7fa2d406db20 auth_method 0 2024-03-29T10:44:46.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:44:46.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.238+0000 7fa2ebfff640 10 client.4526.objecter ms_handle_connect 0x7fa2d406db20 2024-03-29T10:44:46.227 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.245+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2dc050560 con 0x7fa2f413f650 2024-03-29T10:44:46.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.245+0000 7fa2ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:46.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.245+0000 7fa2ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:46.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.560+0000 7fa2f902e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.560+0000 7fa2f902e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.560+0000 7fa2f902e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:46.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.560+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2f4075f10 con 0x7fa2f413f650 2024-03-29T10:44:46.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.561+0000 7fa2ebfff640 1 -- 172.21.15.151:0/3035896074 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fa2dc052490 con 0x7fa2f413f650 2024-03-29T10:44:46.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.561+0000 7fa2ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.561+0000 7fa2ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:46.544 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:46.545 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 10 librados: watch_flush enter 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 10 librados: watch_flush exit 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 >> v1:172.21.15.151:6816/1549873000 conn(0x7fa2d406db20 legacy=0x7fa2d406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 10 monclient: shutdown 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 >> v1:172.21.15.151:6790/0 conn(0x7fa2f413f650 legacy=0x7fa2f41d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 shutdown_connections 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 >> 172.21.15.151:0/3035896074 conn(0x7fa2f4066a00 msgr2=0x7fa2f40b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 shutdown_connections 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 -- 172.21.15.151:0/3035896074 wait complete. 2024-03-29T10:44:46.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:46.564+0000 7fa2f902e640 1 librados: shutdown 2024-03-29T10:44:46.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:53.370 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:53.514 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.530+0000 7fb9e5ea5640 10 monclient: get_monmap_and_config 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.530+0000 7fb9e5ea5640 10 monclient: build_initial_monmap 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.530+0000 7fb9e5ea5640 10 monclient: monmap: 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:44:53.531632+0000 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:44:53.531632+0000 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.531+0000 7fb9e5ea5640 1 Processor -- start 2024-03-29T10:44:53.515 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 1 -- start start 2024-03-29T10:44:53.516 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient: init 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e00a9930 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e0149dc0 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e014ff40 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00aae60 con 0x7fb9e014ff40 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00b4310 con 0x7fb9e00a9930 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00afdd0 con 0x7fb9e0149dc0 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:53.517 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.532+0000 7fb9e5ea5640 10 monclient(hunting): authenticate will time out at 1424.702439s 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9deffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb9e0149dc0 0x7fb9e014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:37808/0 (socket says 172.21.15.151:37808) 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9deffd640 1 -- 172.21.15.151:0/3878557073 learned_addr learned my addr 172.21.15.151:0/3878557073 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734011234 0 0) 0x7fb9e00afdd0 con 0x7fb9e0149dc0 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.518 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.535+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c8003520 con 0x7fb9e0149dc0 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3044722346 0 0) 0x7fb9c8003520 con 0x7fb9e0149dc0 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 >> v1:172.21.15.169:6789/0 conn(0x7fb9e00a9930 legacy=0x7fb9e0143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 >> v1:172.21.15.151:6789/0 conn(0x7fb9e014ff40 legacy=0x7fb9e0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 monclient: found mon.noname-b 2024-03-29T10:44:53.519 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e014a950 con 0x7fb9e0149dc0 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb9d4001a10 con 0x7fb9e0149dc0 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: dump: 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.520 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d4001ca0 con 0x7fb9e0149dc0 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 1 -- 172.21.15.151:0/3878557073 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb9d4002030 con 0x7fb9e0149dc0 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.536+0000 7fb9de7fc640 10 monclient: dump: 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.521 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:53.522 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.537+0000 7fb9e5ea5640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.537+0000 7fb9e5ea5640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.537+0000 7fb9e5ea5640 4 set_mon_vals no callback set 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.538+0000 7fb9e5ea5640 10 monclient: shutdown 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.538+0000 7fb9e5ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.538+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3878557073 >> v1:172.21.15.151:6790/0 conn(0x7fb9e0149dc0 legacy=0x7fb9e014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.538+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3878557073 shutdown_connections 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.538+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3878557073 >> 172.21.15.151:0/3878557073 conn(0x7fb9e00b35e0 msgr2=0x7fb9e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.539+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3878557073 shutdown_connections 2024-03-29T10:44:53.523 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.540+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3878557073 wait complete. 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.541+0000 7fb9e5ea5640 10 monclient: build_initial_monmap 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.541+0000 7fb9e5ea5640 10 monclient: monmap: 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:44:53.524 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.541+0000 7fb9e5ea5640 1 librados: starting msgr at 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.541+0000 7fb9e5ea5640 1 librados: starting objecter 2024-03-29T10:44:53.525 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.542+0000 7fb9e5ea5640 1 Processor -- start 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.542+0000 7fb9e5ea5640 1 -- start start 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.542+0000 7fb9e5ea5640 1 librados: setting wanted keys 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.542+0000 7fb9e5ea5640 1 librados: calling monclient init 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.542+0000 7fb9e5ea5640 10 monclient: init 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.543+0000 7fb9e5ea5640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:53.526 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.543+0000 7fb9e5ea5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e00a9930 addr v1:172.21.15.151:6789/0 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e0149dc0 addr v1:172.21.15.169:6789/0 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e014ff40 addr v1:172.21.15.151:6790/0 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e01dc550 con 0x7fb9e00a9930 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e01db640 con 0x7fb9e0149dc0 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e01db920 con 0x7fb9e014ff40 2024-03-29T10:44:53.531 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.545+0000 7fb9e5ea5640 10 monclient(hunting): authenticate will time out at 1424.714989s 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.547+0000 7fb9df7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb9e00a9930 0x7fb9e011b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:36271/0 (socket says 172.21.15.151:36271) 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.547+0000 7fb9df7fe640 1 -- 172.21.15.151:0/3187577550 learned_addr learned my addr 172.21.15.151:0/3187577550 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 client.?.objecter ms_handle_connect 0x7fb9e014ff40 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270740626 0 0) 0x7fb9e01db920 con 0x7fb9e014ff40 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.532 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.533 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:44:53.533 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.549+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc004eb0 con 0x7fb9e014ff40 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 client.?.objecter ms_handle_connect 0x7fb9e00a9930 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032296508 0 0) 0x7fb9cc004eb0 con 0x7fb9e014ff40 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e01db920 con 0x7fb9e014ff40 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb9d0001750 con 0x7fb9e014ff40 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 monclient(hunting): dump: 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:53.536 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.550+0000 7fb9c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 client.?.objecter ms_handle_connect 0x7fb9e0149dc0 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 518022271 0 0) 0x7fb9e01db920 con 0x7fb9e014ff40 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 >> v1:172.21.15.169:6789/0 conn(0x7fb9e0149dc0 legacy=0x7fb9e01dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 >> v1:172.21.15.151:6789/0 conn(0x7fb9e00a9930 legacy=0x7fb9e011b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.537 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 monclient: found mon.c 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0071c30 con 0x7fb9e014ff40 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 10 monclient: _renew_subs 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e0072410 con 0x7fb9e014ff40 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 10 monclient: _renew_subs 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.538 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e013db70 con 0x7fb9e014ff40 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9e5ea5640 1 librados: init done 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d0001ce0 con 0x7fb9e014ff40 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb9d0003790 con 0x7fb9e014ff40 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9ddffb640 4 set_mon_vals no callback set 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.551+0000 7fb9c7fff640 10 monclient: dump: 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:44:53.539 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:44:53.540 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:44:53.540 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:44:53.540 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:44:53.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.554+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb9d00038f0 con 0x7fb9e014ff40 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fb9d0083710 con 0x7fb9e014ff40 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9c7fff640 10 client.4532.objecter ms_dispatch 0x7fb9e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9df7fe640 10 monclient: get_auth_request con 0x7fb9cc06dbb0 auth_method 0 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9c7fff640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9c7fff640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.555+0000 7fb9c7fff640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T10:44:53.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.558+0000 7fb9c7fff640 10 client.4532.objecter ms_handle_connect 0x7fb9cc06dbb0 2024-03-29T10:44:53.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.559+0000 7fb9e5ea5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.559+0000 7fb9e5ea5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.559+0000 7fb9e5ea5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.559+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0005080 con 0x7fb9e014ff40 2024-03-29T10:44:53.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.570+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9d004ee20 con 0x7fb9e014ff40 2024-03-29T10:44:53.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.570+0000 7fb9c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.570+0000 7fb9c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:53.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.890+0000 7fb9e5ea5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.890+0000 7fb9e5ea5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.890+0000 7fb9e5ea5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:44:53.873 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.890+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9c0005330 con 0x7fb9e014ff40 2024-03-29T10:44:53.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.891+0000 7fb9c7fff640 1 -- 172.21.15.151:0/3187577550 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fb9d0050f20 con 0x7fb9e014ff40 2024-03-29T10:44:53.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.891+0000 7fb9c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.891+0000 7fb9c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:53.874 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:44:53.875 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:53.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 10 librados: watch_flush enter 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 10 librados: watch_flush exit 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 10 client.4532.objecter successfully canceled tick 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 >> v1:172.21.15.151:6816/1549873000 conn(0x7fb9cc06dbb0 legacy=0x7fb9cc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 10 monclient: shutdown 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:53.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.895+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 >> v1:172.21.15.151:6790/0 conn(0x7fb9e014ff40 legacy=0x7fb9e013d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.897+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 shutdown_connections 2024-03-29T10:44:53.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.897+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 >> 172.21.15.151:0/3187577550 conn(0x7fb9e00b35e0 msgr2=0x7fb9e00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:53.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.898+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 shutdown_connections 2024-03-29T10:44:53.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.898+0000 7fb9e5ea5640 1 -- 172.21.15.151:0/3187577550 wait complete. 2024-03-29T10:44:53.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:44:53.898+0000 7fb9e5ea5640 1 librados: shutdown 2024-03-29T10:44:53.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:00.706 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.872+0000 7f499340d640 10 monclient: get_monmap_and_config 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.872+0000 7f499340d640 10 monclient: build_initial_monmap 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.872+0000 7f499340d640 10 monclient: monmap: 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:00.873419+0000 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:00.873419+0000 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:00.856 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:00.857 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:00.857 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.857 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.875+0000 7f499340d640 1 Processor -- start 2024-03-29T10:45:00.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.875+0000 7f499340d640 1 -- start start 2024-03-29T10:45:00.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.875+0000 7f499340d640 10 monclient: init 2024-03-29T10:45:00.858 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.876+0000 7f499340d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:00.859 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.876+0000 7f499340d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:00.859 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.876+0000 7f499340d640 10 monclient(hunting): picked mon.noname-c con 0x7f498c0b4100 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:00.859 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.876+0000 7f499340d640 10 monclient(hunting): picked mon.noname-b con 0x7f498c13ee90 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 10 monclient(hunting): picked mon.noname-a con 0x7f498c0af900 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c0afd10 con 0x7f498c0af900 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c0af6f0 con 0x7f498c0b4100 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c0aaf60 con 0x7f498c13ee90 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f499340d640 10 monclient(hunting): authenticate will time out at 1432.046997s 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f49909a8640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f498c13ee90 0x7f498c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:35320/0 (socket says 172.21.15.151:35320) 2024-03-29T10:45:00.860 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.877+0000 7f49909a8640 1 -- 172.21.15.151:0/68842564 learned_addr learned my addr 172.21.15.151:0/68842564 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550178136 0 0) 0x7f498c0aaf60 con 0x7f498c13ee90 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4978003520 con 0x7f498c13ee90 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.880+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3273890739 0 0) 0x7f4978003520 con 0x7f498c13ee90 2024-03-29T10:45:00.866 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 >> v1:172.21.15.169:6789/0 conn(0x7f498c0b4100 legacy=0x7f498c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 >> v1:172.21.15.151:6789/0 conn(0x7f498c0af900 legacy=0x7f498c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 monclient: found mon.noname-b 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c13e970 con 0x7f498c13ee90 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f49840019f0 con 0x7f498c13ee90 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:00.867 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: dump: 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4984001c50 con 0x7f498c13ee90 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:00.868 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 1 -- 172.21.15.151:0/68842564 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f49840019f0 con 0x7f498c13ee90 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f4983fff640 10 monclient: dump: 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.869 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f499340d640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f499340d640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.881+0000 7f499340d640 4 set_mon_vals no callback set 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.882+0000 7f499340d640 10 monclient: shutdown 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.882+0000 7f499340d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.882+0000 7f499340d640 1 -- 172.21.15.151:0/68842564 >> v1:172.21.15.151:6790/0 conn(0x7f498c13ee90 legacy=0x7f498c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.882+0000 7f499340d640 1 -- 172.21.15.151:0/68842564 shutdown_connections 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.882+0000 7f499340d640 1 -- 172.21.15.151:0/68842564 >> 172.21.15.151:0/68842564 conn(0x7f498c0b35e0 msgr2=0x7f498c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.883+0000 7f499340d640 1 -- 172.21.15.151:0/68842564 shutdown_connections 2024-03-29T10:45:00.870 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.883+0000 7f499340d640 1 -- 172.21.15.151:0/68842564 wait complete. 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.884+0000 7f499340d640 10 monclient: build_initial_monmap 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.884+0000 7f499340d640 10 monclient: monmap: 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:00.874 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.884+0000 7f499340d640 1 librados: starting msgr at 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.884+0000 7f499340d640 1 librados: starting objecter 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.885+0000 7f499340d640 1 Processor -- start 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.885+0000 7f499340d640 1 -- start start 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.885+0000 7f499340d640 1 librados: setting wanted keys 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.885+0000 7f499340d640 1 librados: calling monclient init 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.885+0000 7f499340d640 10 monclient: init 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.886+0000 7f499340d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:00.875 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.886+0000 7f499340d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.886+0000 7f499340d640 10 monclient(hunting): picked mon.noname-c con 0x7f498c0af900 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.886+0000 7f499340d640 10 monclient(hunting): picked mon.noname-a con 0x7f498c0b4100 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 10 monclient(hunting): picked mon.noname-b con 0x7f498c13ee90 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c070e80 con 0x7f498c0b4100 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c071160 con 0x7f498c13ee90 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f49911a9640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f498c0af900 0x7f498c06f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:36592/0 (socket says 172.21.15.151:36592) 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f49911a9640 1 -- 172.21.15.151:0/468839199 learned_addr learned my addr 172.21.15.151:0/468839199 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498c06fd90 con 0x7f498c0af900 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f499340d640 10 monclient(hunting): authenticate will time out at 1432.056605s 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 client.?.objecter ms_handle_connect 0x7f498c0af900 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199692836 0 0) 0x7f498c06fd90 con 0x7f498c0af900 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.876 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4970004eb0 con 0x7f498c0af900 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718641799 0 0) 0x7f4970004eb0 con 0x7f498c0af900 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f498c06fd90 con 0x7f498c0af900 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4988001960 con 0x7f498c0af900 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.887+0000 7f4981ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.888+0000 7f4981ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.888+0000 7f4981ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.877 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.888+0000 7f4981ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.888+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3195078451 0 0) 0x7f498c06fd90 con 0x7f498c0af900 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.888+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 >> v1:172.21.15.169:6789/0 conn(0x7f498c13ee90 legacy=0x7f498c071990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 >> v1:172.21.15.151:6789/0 conn(0x7f498c0b4100 legacy=0x7f498c14efd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 1 monclient: found mon.c 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:00.878 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:00.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c070070 con 0x7f498c0af900 2024-03-29T10:45:00.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:00.880 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.890+0000 7f4981ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 10 monclient: _renew_subs 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c06eb00 con 0x7f498c0af900 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 10 monclient: _renew_subs 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c14f8c0 con 0x7f498c0af900 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.891+0000 7f499340d640 1 librados: init done 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4988001c80 con 0x7f498c0af900 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4988003a40 con 0x7f498c0af900 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.881 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f4981ffb640 10 monclient: dump: 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f49837fe640 4 set_mon_vals no callback set 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.893+0000 7f49777fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.882 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.894+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4988003c60 con 0x7f498c0af900 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.895+0000 7f49777fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.895+0000 7f49777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.895+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c13f3b0 con 0x7f498c0af900 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.895+0000 7f49909a8640 10 monclient: get_auth_request con 0x7f497006db80 auth_method 0 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4988054940 con 0x7f498c0af900 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 10 client.4538.objecter ms_dispatch 0x7f498c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T10:45:00.883 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.896+0000 7f4981ffb640 10 client.4538.objecter ms_handle_connect 0x7f497006db80 2024-03-29T10:45:00.889 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.907+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4988050450 con 0x7f498c0af900 2024-03-29T10:45:00.889 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.907+0000 7f4981ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.890 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:00.907+0000 7f4981ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:01.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.231+0000 7f49777fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:01.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.231+0000 7f49777fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:01.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.231+0000 7f49777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:01.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.231+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f498c075f10 con 0x7f498c0af900 2024-03-29T10:45:01.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.232+0000 7f4981ffb640 1 -- 172.21.15.151:0/468839199 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4988052380 con 0x7f498c0af900 2024-03-29T10:45:01.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.232+0000 7f4981ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:01.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.232+0000 7f4981ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:01.215 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:01.216 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.236+0000 7f49777fe640 10 librados: watch_flush enter 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.236+0000 7f49777fe640 10 librados: watch_flush exit 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.236+0000 7f49777fe640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.236+0000 7f49777fe640 10 client.4538.objecter successfully canceled tick 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 >> v1:172.21.15.151:6816/1549873000 conn(0x7f497006db80 legacy=0x7f4970070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:01.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 10 monclient: shutdown 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 >> v1:172.21.15.151:6790/0 conn(0x7f498c0af900 legacy=0x7f498c06f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 shutdown_connections 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 >> 172.21.15.151:0/468839199 conn(0x7f498c0b35e0 msgr2=0x7f498c0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 shutdown_connections 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.237+0000 7f49777fe640 1 -- 172.21.15.151:0/468839199 wait complete. 2024-03-29T10:45:01.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:01.238+0000 7f49777fe640 1 librados: shutdown 2024-03-29T10:45:01.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:08.045 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.209+0000 7f8ea759e640 10 monclient: get_monmap_and_config 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.209+0000 7f8ea759e640 10 monclient: build_initial_monmap 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.209+0000 7f8ea759e640 10 monclient: monmap: 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.192 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:08.210330+0000 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:08.210330+0000 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.193 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.210+0000 7f8ea759e640 1 Processor -- start 2024-03-29T10:45:08.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.210+0000 7f8ea759e640 1 -- start start 2024-03-29T10:45:08.194 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.210+0000 7f8ea759e640 10 monclient: init 2024-03-29T10:45:08.196 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea00389e0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea0038dc0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea0030660 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00c54c0 con 0x7f8ea00389e0 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea0036430 con 0x7f8ea0038dc0 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea0039840 con 0x7f8ea0030660 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.211+0000 7f8ea759e640 10 monclient(hunting): authenticate will time out at 1439.381057s 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea6d9d640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f8ea0030660 0x7f8ea00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:45113/0 (socket says 172.21.15.151:45113) 2024-03-29T10:45:08.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea6d9d640 1 -- 172.21.15.151:0/3949281241 learned_addr learned my addr 172.21.15.151:0/3949281241 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:08.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997070864 0 0) 0x7f8ea0039840 con 0x7f8ea0030660 2024-03-29T10:45:08.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.214+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90003520 con 0x7f8ea0030660 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.215+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051998349 0 0) 0x7f8e90003520 con 0x7f8ea0030660 2024-03-29T10:45:08.199 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.215+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 >> v1:172.21.15.169:6789/0 conn(0x7f8ea0038dc0 legacy=0x7f8ea00cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 >> v1:172.21.15.151:6789/0 conn(0x7f8ea00389e0 legacy=0x7f8ea0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 1 monclient: found mon.noname-b 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea0035690 con 0x7f8ea0030660 2024-03-29T10:45:08.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: _finish_auth 0 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8ea804f730 con 0x7f8ea0030660 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea559a640 10 monclient: dump: 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:08.203 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:08.204 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea759e640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:45:08.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.219+0000 7f8ea759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea8075f10 con 0x7f8ea0030660 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 1 -- 172.21.15.151:0/3949281241 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8ea80758b0 con 0x7f8ea0030660 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea559a640 10 monclient: dump: 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.205 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.220+0000 7f8ea759e640 4 set_mon_vals no callback set 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 10 monclient: shutdown 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 1 -- 172.21.15.151:0/3949281241 >> v1:172.21.15.151:6790/0 conn(0x7f8ea0030660 legacy=0x7f8ea00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 1 -- 172.21.15.151:0/3949281241 shutdown_connections 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 1 -- 172.21.15.151:0/3949281241 >> 172.21.15.151:0/3949281241 conn(0x7f8ea0005c00 msgr2=0x7f8ea0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 1 -- 172.21.15.151:0/3949281241 shutdown_connections 2024-03-29T10:45:08.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.221+0000 7f8ea759e640 1 -- 172.21.15.151:0/3949281241 wait complete. 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.222+0000 7f8ea759e640 10 monclient: build_initial_monmap 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.222+0000 7f8ea759e640 10 monclient: monmap: 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:08.209 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 librados: starting msgr at 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 librados: starting objecter 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 Processor -- start 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 -- start start 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 librados: setting wanted keys 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 1 librados: calling monclient init 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.223+0000 7f8ea759e640 10 monclient: init 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea0030660 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea00389e0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea000ce50 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:08.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00098b0 con 0x7f8ea00389e0 2024-03-29T10:45:08.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea0164740 con 0x7f8ea0030660 2024-03-29T10:45:08.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea00c3410 con 0x7f8ea000ce50 2024-03-29T10:45:08.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:08.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.224+0000 7f8ea759e640 10 monclient(hunting): authenticate will time out at 1439.394360s 2024-03-29T10:45:08.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.228+0000 7f8ea659c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f8ea0030660 0x7f8ea0164bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:36959/0 (socket says 172.21.15.151:36959) 2024-03-29T10:45:08.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.228+0000 7f8ea659c640 1 -- 172.21.15.151:0/1073233077 learned_addr learned my addr 172.21.15.151:0/1073233077 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:08.214 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.231+0000 7f8e96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ea0030660 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.231+0000 7f8e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.231+0000 7f8e96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ea00389e0 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.231+0000 7f8e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ea000ce50 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600405331 0 0) 0x7f8ea00c3410 con 0x7f8ea000ce50 2024-03-29T10:45:08.215 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.232+0000 7f8e96ffd640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e88004eb0 con 0x7f8ea000ce50 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698117932 0 0) 0x7f8ea00098b0 con 0x7f8ea00389e0 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:45:08.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea00c3410 con 0x7f8ea00389e0 2024-03-29T10:45:08.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314087003 0 0) 0x7f8e88004eb0 con 0x7f8ea000ce50 2024-03-29T10:45:08.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea00098b0 con 0x7f8ea000ce50 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8ea804f400 con 0x7f8ea000ce50 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.219 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754011291 0 0) 0x7f8ea00c3410 con 0x7f8ea00389e0 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e88004eb0 con 0x7f8ea00389e0 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.233+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8e9c001750 con 0x7f8ea00389e0 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:08.220 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4114315038 0 0) 0x7f8ea00098b0 con 0x7f8ea000ce50 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 >> v1:172.21.15.169:6789/0 conn(0x7f8ea0030660 legacy=0x7f8ea0164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 >> v1:172.21.15.151:6789/0 conn(0x7f8ea00389e0 legacy=0x7f8ea00c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 1 monclient: found mon.c 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea0164410 con 0x7f8ea000ce50 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 10 monclient: _renew_subs 2024-03-29T10:45:08.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea01652e0 con 0x7f8ea000ce50 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 10 monclient: _renew_subs 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea000d8f0 con 0x7f8ea000ce50 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.234+0000 7f8ea759e640 1 librados: init done 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.235+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea804fdd0 con 0x7f8ea000ce50 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.235+0000 7f8e96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:08.222 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.235+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8ea8067bf0 con 0x7f8ea000ce50 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.236+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.236+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.236+0000 7f8e96ffd640 10 monclient: dump: 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:08.224 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.236+0000 7f8ea759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.236+0000 7f8ea4d99640 4 set_mon_vals no callback set 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.237+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f8ea8067d50 con 0x7f8ea000ce50 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.237+0000 7f8ea759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8ea759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea0007b70 con 0x7f8ea000ce50 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f8ea80ff340 con 0x7f8ea000ce50 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8e96ffd640 10 client.4544.objecter ms_dispatch 0x7f8ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8e96ffd640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8e96ffd640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.238+0000 7f8ea659c640 10 monclient: get_auth_request con 0x7f8e8806de50 auth_method 0 2024-03-29T10:45:08.225 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.239+0000 7f8e96ffd640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T10:45:08.226 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.239+0000 7f8e96ffd640 10 client.4544.objecter ms_handle_connect 0x7f8e8806de50 2024-03-29T10:45:08.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.245+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ea80cac20 con 0x7f8ea000ce50 2024-03-29T10:45:08.228 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.246+0000 7f8e96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:08.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.247+0000 7f8e96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:08.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.570+0000 7f8ea759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.570+0000 7f8ea759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.570+0000 7f8ea759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:08.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.570+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ea0005120 con 0x7f8ea000ce50 2024-03-29T10:45:08.556 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.574+0000 7f8e96ffd640 1 -- 172.21.15.151:0/1073233077 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f8ea80ccb50 con 0x7f8ea000ce50 2024-03-29T10:45:08.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.574+0000 7f8e96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.574+0000 7f8e96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:08.559 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:08.559 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 10 librados: watch_flush enter 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 10 librados: watch_flush exit 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 10 client.4544.objecter successfully canceled tick 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 >> v1:172.21.15.151:6816/1549873000 conn(0x7f8e8806de50 legacy=0x7f8e88070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 10 monclient: shutdown 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 >> v1:172.21.15.151:6790/0 conn(0x7f8ea000ce50 legacy=0x7f8ea00c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 shutdown_connections 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 >> 172.21.15.151:0/1073233077 conn(0x7f8ea0005c00 msgr2=0x7f8ea0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.580+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 shutdown_connections 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.581+0000 7f8ea759e640 1 -- 172.21.15.151:0/1073233077 wait complete. 2024-03-29T10:45:08.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:08.581+0000 7f8ea759e640 1 librados: shutdown 2024-03-29T10:45:08.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:15.386 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:15.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.556+0000 7f7c3d932640 10 monclient: get_monmap_and_config 2024-03-29T10:45:15.539 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.556+0000 7f7c3d932640 10 monclient: build_initial_monmap 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.556+0000 7f7c3d932640 10 monclient: monmap: 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:15.557868+0000 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:15.557868+0000 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:15.540 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.558+0000 7f7c3d932640 1 Processor -- start 2024-03-29T10:45:15.541 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.558+0000 7f7c3d932640 1 -- start start 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.558+0000 7f7c3d932640 10 monclient: init 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-c con 0x7f7c380b1f60 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-a con 0x7f7c38071eb0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-b con 0x7f7c38143d10 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3806ef80 con 0x7f7c38071eb0 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3806f260 con 0x7f7c380b1f60 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c380b40c0 con 0x7f7c38143d10 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:15.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.559+0000 7f7c3d932640 10 monclient(hunting): authenticate will time out at 1446.728842s 2024-03-29T10:45:15.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.561+0000 7f7c367fc640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7c38071eb0 0x7f7c38146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37626/0 (socket says 172.21.15.151:37626) 2024-03-29T10:45:15.544 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.561+0000 7f7c367fc640 1 -- 172.21.15.151:0/3857364092 learned_addr learned my addr 172.21.15.151:0/3857364092 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909786454 0 0) 0x7f7c3806ef80 con 0x7f7c38071eb0 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:45:15.545 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.562+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c20003520 con 0x7f7c38071eb0 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168887263 0 0) 0x7f7c20003520 con 0x7f7c38071eb0 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 >> v1:172.21.15.151:6790/0 conn(0x7f7c38143d10 legacy=0x7f7c38146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 >> v1:172.21.15.169:6789/0 conn(0x7f7c380b1f60 legacy=0x7f7c38143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 monclient: found mon.noname-a 2024-03-29T10:45:15.546 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38155d70 con 0x7f7c38071eb0 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c28001690 con 0x7f7c38071eb0 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c35ffb640 10 monclient: dump: 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.547 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c3d932640 5 monclient: authenticate success, global_id 4485 2024-03-29T10:45:15.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.563+0000 7f7c3d932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c28001860 con 0x7f7c38071eb0 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 1 -- 172.21.15.151:0/3857364092 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c28001cb0 con 0x7f7c38071eb0 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:15.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c35ffb640 10 monclient: dump: 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c3d932640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.566+0000 7f7c3d932640 4 set_mon_vals no callback set 2024-03-29T10:45:15.550 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.567+0000 7f7c3d932640 10 monclient: shutdown 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.567+0000 7f7c3d932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.567+0000 7f7c3d932640 1 -- 172.21.15.151:0/3857364092 >> v1:172.21.15.151:6789/0 conn(0x7f7c38071eb0 legacy=0x7f7c38146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.567+0000 7f7c3d932640 1 -- 172.21.15.151:0/3857364092 shutdown_connections 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.567+0000 7f7c3d932640 1 -- 172.21.15.151:0/3857364092 >> 172.21.15.151:0/3857364092 conn(0x7f7c38066a00 msgr2=0x7f7c38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.568+0000 7f7c3d932640 1 -- 172.21.15.151:0/3857364092 shutdown_connections 2024-03-29T10:45:15.551 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.568+0000 7f7c3d932640 1 -- 172.21.15.151:0/3857364092 wait complete. 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.569+0000 7f7c3d932640 10 monclient: build_initial_monmap 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.569+0000 7f7c3d932640 10 monclient: monmap: 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:15.552 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.569+0000 7f7c3d932640 1 librados: starting msgr at 2024-03-29T10:45:15.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.569+0000 7f7c3d932640 1 librados: starting objecter 2024-03-29T10:45:15.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.569+0000 7f7c3d932640 1 Processor -- start 2024-03-29T10:45:15.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.572+0000 7f7c3d932640 1 -- start start 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.572+0000 7f7c3d932640 1 librados: setting wanted keys 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.572+0000 7f7c3d932640 1 librados: calling monclient init 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.572+0000 7f7c3d932640 10 monclient: init 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-c con 0x7f7c380b1f60 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-b con 0x7f7c38143d10 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient(hunting): picked mon.noname-a con 0x7f7c381e70c0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c380ac160 con 0x7f7c381e70c0 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c380ac440 con 0x7f7c38143d10 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c380aa5d0 con 0x7f7c380b1f60 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:15.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.573+0000 7f7c3d932640 10 monclient(hunting): authenticate will time out at 1446.742853s 2024-03-29T10:45:15.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.575+0000 7f7c377fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f7c381e70c0 0x7f7c3813fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:38650/0 (socket says 172.21.15.151:38650) 2024-03-29T10:45:15.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.575+0000 7f7c377fe640 1 -- 172.21.15.151:0/1136017213 learned_addr learned my addr 172.21.15.151:0/1136017213 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 client.?.objecter ms_handle_connect 0x7f7c381e70c0 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275492922 0 0) 0x7f7c380ac160 con 0x7f7c381e70c0 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.576+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c24004eb0 con 0x7f7c381e70c0 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.577+0000 7f7c1ffff640 10 client.?.objecter ms_handle_connect 0x7f7c380b1f60 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.577+0000 7f7c1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1267998461 0 0) 0x7f7c24004eb0 con 0x7f7c381e70c0 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c24005300 con 0x7f7c381e70c0 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c300042a0 con 0x7f7c381e70c0 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 10 monclient(hunting): dump: 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:15.565 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.578+0000 7f7c1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.579+0000 7f7c367fc640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.169:6789/0 conn(0x7f7c38143d10 legacy=0x7f7c380aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.579+0000 7f7c367fc640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.169:6789/0 conn(0x7f7c38143d10 legacy=0x7f7c380aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:45:15.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.579+0000 7f7c367fc640 1 --1- 172.21.15.151:0/1136017213 >> v1:172.21.15.169:6789/0 conn(0x7f7c38143d10 0x7f7c380aab40 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.579+0000 7f7c367fc640 1 --1- 172.21.15.151:0/1136017213 >> v1:172.21.15.169:6789/0 conn(0x7f7c38143d10 0x7f7c380aab40 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.579+0000 7f7c1ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.169:6789/0 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99529571 0 0) 0x7f7c380aa5d0 con 0x7f7c380b1f60 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.580+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c24005850 con 0x7f7c380b1f60 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3187629862 0 0) 0x7f7c24005300 con 0x7f7c381e70c0 2024-03-29T10:45:15.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.151:6790/0 conn(0x7f7c380b1f60 legacy=0x7f7c380a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.169:6789/0 conn(0x7f7c38143d10 legacy=0x7f7c380aab40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 1 monclient: found mon.a 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380ab250 con 0x7f7c381e70c0 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 10 monclient: _finish_auth 0 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:15.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 5 monclient: authenticate success, global_id 4488 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 10 monclient: _renew_subs 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c381429c0 con 0x7f7c381e70c0 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 10 monclient: _renew_subs 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c38140360 con 0x7f7c381e70c0 2024-03-29T10:45:15.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.581+0000 7f7c3d932640 1 librados: init done 2024-03-29T10:45:15.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.583+0000 7f7c3d932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.583+0000 7f7c3d932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.583+0000 7f7c3d932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.583+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c380ab440 con 0x7f7c381e70c0 2024-03-29T10:45:15.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c30004830 con 0x7f7c381e70c0 2024-03-29T10:45:15.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:15.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7c300042a0 con 0x7f7c381e70c0 2024-03-29T10:45:15.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.585+0000 7f7c1ffff640 10 monclient: dump: 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:15.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.586+0000 7f7c34ff9640 4 set_mon_vals no callback set 2024-03-29T10:45:15.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f7c30001250 con 0x7f7c381e70c0 2024-03-29T10:45:15.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f7c300545a0 con 0x7f7c381e70c0 2024-03-29T10:45:15.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 10 client.4488.objecter ms_dispatch 0x7f7c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:15.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:15.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:15.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.591+0000 7f7c1ffff640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T10:45:15.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.592+0000 7f7c36ffd640 10 monclient: get_auth_request con 0x7f7c2406e280 auth_method 0 2024-03-29T10:45:15.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.593+0000 7f7c1ffff640 10 client.4488.objecter ms_handle_connect 0x7f7c2406e280 2024-03-29T10:45:15.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.601+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c30050840 con 0x7f7c381e70c0 2024-03-29T10:45:15.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.601+0000 7f7c1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.584 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.601+0000 7f7c1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:15.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.996+0000 7f7c3d932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.996+0000 7f7c3d932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.996+0000 7f7c3d932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.996+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c380aa860 con 0x7f7c381e70c0 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.997+0000 7f7c1ffff640 1 -- 172.21.15.151:0/1136017213 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f7c380aa860 con 0x7f7c381e70c0 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.997+0000 7f7c1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:15.997+0000 7f7c1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:15.982 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:15.982 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 10 librados: watch_flush enter 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 10 librados: watch_flush exit 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 10 client.4488.objecter successfully canceled tick 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.151:6816/1549873000 conn(0x7f7c2406e280 legacy=0x7f7c24070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 10 monclient: shutdown 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:15.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.003+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 >> v1:172.21.15.151:6789/0 conn(0x7f7c381e70c0 legacy=0x7f7c3813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.006+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 shutdown_connections 2024-03-29T10:45:15.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.006+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 >> 172.21.15.151:0/1136017213 conn(0x7f7c38066a00 msgr2=0x7f7c380b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:15.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.006+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 shutdown_connections 2024-03-29T10:45:15.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.006+0000 7f7c3d932640 1 -- 172.21.15.151:0/1136017213 wait complete. 2024-03-29T10:45:15.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:16.010+0000 7f7c3d932640 1 librados: shutdown 2024-03-29T10:45:16.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:22.816 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.982+0000 7f2f77d56640 10 monclient: get_monmap_and_config 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.982+0000 7f2f77d56640 10 monclient: build_initial_monmap 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.983+0000 7f2f77d56640 10 monclient: monmap: 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:22.983945+0000 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:22.983945+0000 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:22.967 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.984+0000 7f2f77d56640 1 Processor -- start 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.984+0000 7f2f77d56640 1 -- start start 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.984+0000 7f2f77d56640 10 monclient: init 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:22.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-b con 0x7f2f700a9ac0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-a con 0x7f2f7013ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-c con 0x7f2f700af900 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700afd10 con 0x7f2f7013ee90 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700af6f0 con 0x7f2f700af900 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700aac80 con 0x7f2f700a9ac0 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f77d56640 10 monclient(hunting): authenticate will time out at 1454.155146s 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f75af2640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f2f700a9ac0 0x7f2f700adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56552/0 (socket says 172.21.15.151:56552) 2024-03-29T10:45:22.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.985+0000 7f2f75af2640 1 -- 172.21.15.151:0/1599379767 learned_addr learned my addr 172.21.15.151:0/1599379767 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268605191 0 0) 0x7f2f700aac80 con 0x7f2f700a9ac0 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 10 monclient(hunting): _negotiate 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003520 con 0x7f2f700a9ac0 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1139485424 0 0) 0x7f2f5c003520 con 0x7f2f700a9ac0 2024-03-29T10:45:22.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.986+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 >> v1:172.21.15.169:6789/0 conn(0x7f2f700af900 legacy=0x7f2f70149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:22.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.988+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 >> v1:172.21.15.151:6789/0 conn(0x7f2f7013ee90 legacy=0x7f2f7013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:22.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.990+0000 7f2f74af0640 10 monclient: _finish_hunting 0 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.990+0000 7f2f74af0640 1 monclient: found mon.noname-b 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.990+0000 7f2f74af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.990+0000 7f2f74af0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.990+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700aaf60 con 0x7f2f700a9ac0 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: _finish_auth 0 2024-03-29T10:45:22.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: _check_auth_tickets 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f77d56640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f77d56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2f68001a10 con 0x7f2f700a9ac0 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: dump: 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.977 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f77d56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f68001ca0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 1 -- 172.21.15.151:0/1599379767 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2f68002030 con 0x7f2f700a9ac0 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f74af0640 10 monclient: dump: 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:22.979 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f77d56640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.991+0000 7f2f77d56640 4 set_mon_vals no callback set 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.992+0000 7f2f77d56640 10 monclient: shutdown 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.992+0000 7f2f77d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.992+0000 7f2f77d56640 1 -- 172.21.15.151:0/1599379767 >> v1:172.21.15.151:6790/0 conn(0x7f2f700a9ac0 legacy=0x7f2f700adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.993+0000 7f2f77d56640 1 -- 172.21.15.151:0/1599379767 shutdown_connections 2024-03-29T10:45:22.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.993+0000 7f2f77d56640 1 -- 172.21.15.151:0/1599379767 >> 172.21.15.151:0/1599379767 conn(0x7f2f700b35e0 msgr2=0x7f2f700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:22.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.993+0000 7f2f77d56640 1 -- 172.21.15.151:0/1599379767 shutdown_connections 2024-03-29T10:45:22.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.993+0000 7f2f77d56640 1 -- 172.21.15.151:0/1599379767 wait complete. 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.994+0000 7f2f77d56640 10 monclient: build_initial_monmap 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.994+0000 7f2f77d56640 10 monclient: monmap: 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:22.985 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.994+0000 7f2f77d56640 1 librados: starting msgr at 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.994+0000 7f2f77d56640 1 librados: starting objecter 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.995+0000 7f2f77d56640 1 Processor -- start 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.995+0000 7f2f77d56640 1 -- start start 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.995+0000 7f2f77d56640 1 librados: setting wanted keys 2024-03-29T10:45:22.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.995+0000 7f2f77d56640 1 librados: calling monclient init 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.995+0000 7f2f77d56640 10 monclient: init 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.996+0000 7f2f77d56640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.996+0000 7f2f77d56640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.996+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-c con 0x7f2f700a9ac0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.996+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-a con 0x7f2f700af900 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.999+0000 7f2f75af2640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f2f700a9ac0 0x7f2f70135030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:32854/0 (socket says 172.21.15.151:32854) 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.999+0000 7f2f75af2640 1 -- 172.21.15.151:0/2809205470 learned_addr learned my addr 172.21.15.151:0/2809205470 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.999+0000 7f2f77d56640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7013ee90 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.999+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f70140b00 con 0x7f2f700af900 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:22.999+0000 7f2f667fc640 10 client.?.objecter ms_handle_connect 0x7f2f700a9ac0 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:22.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7013d7e0 con 0x7f2f7013ee90 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7013dac0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f77d56640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f77d56640 10 monclient(hunting): authenticate will time out at 1454.170119s 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224784372 0 0) 0x7f2f7013dac0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.000+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f54004eb0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.988 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 client.?.objecter ms_handle_connect 0x7f2f7013ee90 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422993646 0 0) 0x7f2f7013d7e0 con 0x7f2f7013ee90 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f7013dac0 con 0x7f2f7013ee90 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4023145781 0 0) 0x7f2f54004eb0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.989 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f7013d7e0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2f68001950 con 0x7f2f700a9ac0 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): dump: 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:22.990 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:22.991 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:22.991 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.002+0000 7f2f667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:22.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036500891 0 0) 0x7f2f7013d7e0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 >> v1:172.21.15.169:6789/0 conn(0x7f2f7013ee90 legacy=0x7f2f701e25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 >> v1:172.21.15.151:6789/0 conn(0x7f2f700af900 legacy=0x7f2f701da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 1 monclient: found mon.c 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f70140d30 con 0x7f2f700a9ac0 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:45:22.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 10 monclient: _renew_subs 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7013fa80 con 0x7f2f700a9ac0 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 10 monclient: _renew_subs 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f70135740 con 0x7f2f700a9ac0 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.003+0000 7f2f77d56640 1 librados: init done 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.005+0000 7f2f77d56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.005+0000 7f2f77d56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.005+0000 7f2f77d56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:22.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.005+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70075f10 con 0x7f2f700a9ac0 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f680015f0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2f68004fb0 con 0x7f2f700a9ac0 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f667fc640 10 monclient: dump: 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.995 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:22.996 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.006+0000 7f2f67fff640 4 set_mon_vals no callback set 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.010+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f2f6801af80 con 0x7f2f700a9ac0 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f2f68084e80 con 0x7f2f700a9ac0 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f667fc640 10 client.4553.objecter ms_dispatch 0x7f2f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f667fc640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f667fc640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f667fc640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.011+0000 7f2f752f1640 10 monclient: get_auth_request con 0x7f2f5406dd50 auth_method 0 2024-03-29T10:45:22.997 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.012+0000 7f2f667fc640 10 client.4553.objecter ms_handle_connect 0x7f2f5406dd50 2024-03-29T10:45:23.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.018+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f68050590 con 0x7f2f700a9ac0 2024-03-29T10:45:23.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.018+0000 7f2f667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:23.001 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.018+0000 7f2f667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:23.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.342+0000 7f2f77d56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.342+0000 7f2f77d56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.342+0000 7f2f77d56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:23.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.342+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f7006ab70 con 0x7f2f700a9ac0 2024-03-29T10:45:23.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.343+0000 7f2f667fc640 1 -- 172.21.15.151:0/2809205470 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f2f68052690 con 0x7f2f700a9ac0 2024-03-29T10:45:23.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.343+0000 7f2f667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.343+0000 7f2f667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:23.327 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:23.327 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:23.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.347+0000 7f2f77d56640 10 librados: watch_flush enter 2024-03-29T10:45:23.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 10 librados: watch_flush exit 2024-03-29T10:45:23.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 10 client.4553.objecter successfully canceled tick 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 >> v1:172.21.15.151:6816/1549873000 conn(0x7f2f5406dd50 legacy=0x7f2f54070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 10 monclient: shutdown 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 >> v1:172.21.15.151:6790/0 conn(0x7f2f700a9ac0 legacy=0x7f2f70135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 shutdown_connections 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.348+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 >> 172.21.15.151:0/2809205470 conn(0x7f2f700b35e0 msgr2=0x7f2f700adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.349+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 shutdown_connections 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.349+0000 7f2f77d56640 1 -- 172.21.15.151:0/2809205470 wait complete. 2024-03-29T10:45:23.332 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:23.349+0000 7f2f77d56640 1 librados: shutdown 2024-03-29T10:45:23.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:30.157 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:30.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.311+0000 7f64668d9640 10 monclient: get_monmap_and_config 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.311+0000 7f64668d9640 10 monclient: build_initial_monmap 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.311+0000 7f64668d9640 10 monclient: monmap: 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:30.312103+0000 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:30.312103+0000 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:30.294 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.312+0000 7f64668d9640 1 Processor -- start 2024-03-29T10:45:30.295 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.312+0000 7f64668d9640 1 -- start start 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.312+0000 7f64668d9640 10 monclient: init 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-a con 0x7f64601457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-b con 0x7f646013f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:30.296 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f645ffff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f64601457f0 0x7f646013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45610/0 (socket says 172.21.15.151:45610) 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-c con 0x7f64600a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64600aaef0 con 0x7f64601457f0 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460145bd0 con 0x7f64600a9d70 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460145eb0 con 0x7f646013f650 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f64668d9640 10 monclient(hunting): authenticate will time out at 1461.483397s 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.313+0000 7f645ffff640 1 -- 172.21.15.151:0/622904829 learned_addr learned my addr 172.21.15.151:0/622904829 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963692578 0 0) 0x7f64600aaef0 con 0x7f64601457f0 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.297 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6448003520 con 0x7f64601457f0 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 230621267 0 0) 0x7f6448003520 con 0x7f64601457f0 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 >> v1:172.21.15.151:6790/0 conn(0x7f646013f650 legacy=0x7f64600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:30.298 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.314+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 >> v1:172.21.15.169:6789/0 conn(0x7f64600a9d70 legacy=0x7f646014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 1 monclient: found mon.noname-a 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460146340 con 0x7f64601457f0 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: _finish_auth 0 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6450001690 con 0x7f64601457f0 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.300 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f645effd640 10 monclient: dump: 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f64668d9640 5 monclient: authenticate success, global_id 4491 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.316+0000 7f64668d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6450001860 con 0x7f64601457f0 2024-03-29T10:45:30.301 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 1 -- 172.21.15.151:0/622904829 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6450001cb0 con 0x7f64601457f0 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f645effd640 10 monclient: dump: 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:30.302 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f64668d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:30.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.317+0000 7f64668d9640 4 set_mon_vals no callback set 2024-03-29T10:45:30.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.318+0000 7f64668d9640 10 monclient: shutdown 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.318+0000 7f64668d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.318+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 >> v1:172.21.15.151:6789/0 conn(0x7f64601457f0 legacy=0x7f646013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.318+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 shutdown_connections 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.318+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 >> 172.21.15.151:0/622904829 conn(0x7f6460066a00 msgr2=0x7f6460066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.319+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 shutdown_connections 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.319+0000 7f64668d9640 1 -- 172.21.15.151:0/622904829 wait complete. 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.319+0000 7f64668d9640 10 monclient: build_initial_monmap 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.319+0000 7f64668d9640 10 monclient: monmap: 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:30.309 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.320+0000 7f64668d9640 1 librados: starting msgr at 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.320+0000 7f64668d9640 1 librados: starting objecter 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.320+0000 7f64668d9640 1 Processor -- start 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.320+0000 7f64668d9640 1 -- start start 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.321+0000 7f64668d9640 1 librados: setting wanted keys 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.321+0000 7f64668d9640 1 librados: calling monclient init 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.321+0000 7f64668d9640 10 monclient: init 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f64668d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f64668d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:30.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-a con 0x7f64600a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f645ffff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f64600a9d70 0x7f6460144a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:46234/0 (socket says 172.21.15.151:46234) 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f645ffff640 1 -- 172.21.15.151:0/2701065760 learned_addr learned my addr 172.21.15.151:0/2701065760 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f645cff9640 10 client.?.objecter ms_handle_connect 0x7f64600a9d70 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f645cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.322+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-b con 0x7f646013f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 10 monclient(hunting): picked mon.noname-c con 0x7f64601457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460140b70 con 0x7f64600a9d70 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460140da0 con 0x7f646013f650 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6460143320 con 0x7f64601457f0 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f64668d9640 10 monclient(hunting): authenticate will time out at 1461.493143s 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f645cff9640 10 client.?.objecter ms_handle_connect 0x7f64601457f0 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f645cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:30.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288752935 0 0) 0x7f6460143320 con 0x7f64601457f0 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f645cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.323+0000 7f645cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c004eb0 con 0x7f64601457f0 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447309128 0 0) 0x7f6460140b70 con 0x7f64600a9d70 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460143320 con 0x7f64600a9d70 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143957223 0 0) 0x7f644c004eb0 con 0x7f64601457f0 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.324+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6460140b70 con 0x7f64601457f0 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6458001b70 con 0x7f64601457f0 2024-03-29T10:45:30.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): dump: 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1320723179 0 0) 0x7f6460143320 con 0x7f64600a9d70 2024-03-29T10:45:30.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f644c004eb0 con 0x7f64600a9d70 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6450001b70 con 0x7f64600a9d70 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.325+0000 7f645cff9640 10 monclient(hunting): dump: 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:30.314 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2500680895 0 0) 0x7f6460140b70 con 0x7f64601457f0 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 >> v1:172.21.15.169:6789/0 conn(0x7f646013f650 legacy=0x7f6460134620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 >> v1:172.21.15.151:6789/0 conn(0x7f64600a9d70 legacy=0x7f6460144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 monclient: found mon.c 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64601441c0 con 0x7f64601457f0 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:30.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 10 monclient: _renew_subs 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6460145170 con 0x7f64601457f0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 10 monclient: _renew_subs 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64601413a0 con 0x7f64601457f0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6458001cd0 con 0x7f64601457f0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.326+0000 7f64668d9640 1 librados: init done 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6458003bb0 con 0x7f64601457f0 2024-03-29T10:45:30.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645cff9640 10 monclient: dump: 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:30.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.327+0000 7f645e7fc640 4 set_mon_vals no callback set 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.329+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f6458019b80 con 0x7f64601457f0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.329+0000 7f64668d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.329+0000 7f64668d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.329+0000 7f64668d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.329+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6440005080 con 0x7f64601457f0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f6458083a80 con 0x7f64601457f0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 10 client.4556.objecter ms_dispatch 0x7f6460000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645ffff640 10 monclient: get_auth_request con 0x7f644c06df00 auth_method 0 2024-03-29T10:45:30.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.330+0000 7f645cff9640 10 client.4556.objecter ms_handle_connect 0x7f644c06df00 2024-03-29T10:45:30.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.337+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f645804f360 con 0x7f64601457f0 2024-03-29T10:45:30.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.337+0000 7f645cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.337+0000 7f645cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:30.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.649+0000 7f64668d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.649+0000 7f64668d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.649+0000 7f64668d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.649+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6440005330 con 0x7f64601457f0 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.650+0000 7f645cff9640 1 -- 172.21.15.151:0/2701065760 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f6458051290 con 0x7f64601457f0 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.650+0000 7f645cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.650+0000 7f645cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:30.633 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:30.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 10 librados: watch_flush enter 2024-03-29T10:45:30.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 10 librados: watch_flush exit 2024-03-29T10:45:30.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T10:45:30.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 10 client.4556.objecter successfully canceled tick 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 >> v1:172.21.15.151:6816/1549873000 conn(0x7f644c06df00 legacy=0x7f644c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 10 monclient: shutdown 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 >> v1:172.21.15.151:6790/0 conn(0x7f64601457f0 legacy=0x7f6460134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 shutdown_connections 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.653+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 >> 172.21.15.151:0/2701065760 conn(0x7f6460066a00 msgr2=0x7f646013c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.654+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 shutdown_connections 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.654+0000 7f64668d9640 1 -- 172.21.15.151:0/2701065760 wait complete. 2024-03-29T10:45:30.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:30.654+0000 7f64668d9640 1 librados: shutdown 2024-03-29T10:45:30.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:37.458 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:37.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.621+0000 7f9331cff640 10 monclient: get_monmap_and_config 2024-03-29T10:45:37.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.621+0000 7f9331cff640 10 monclient: build_initial_monmap 2024-03-29T10:45:37.604 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.621+0000 7f9331cff640 10 monclient: monmap: 2024-03-29T10:45:37.604 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:37.604 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:37.622622+0000 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:37.622622+0000 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.605 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.622+0000 7f9331cff640 1 Processor -- start 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.623+0000 7f9331cff640 1 -- start start 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.623+0000 7f9331cff640 10 monclient: init 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-b con 0x7f932c0b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-a con 0x7f932c14a700 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-c con 0x7f932c0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0aba70 con 0x7f932c14a700 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0aecb0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0af150 con 0x7f932c0b0770 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f9331cff640 10 monclient(hunting): authenticate will time out at 1468.793803s 2024-03-29T10:45:37.607 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f932affd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f932c14a700 0x7f932c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:55574/0 (socket says 172.21.15.151:55574) 2024-03-29T10:45:37.608 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.624+0000 7f932affd640 1 -- 172.21.15.151:0/2589793317 learned_addr learned my addr 172.21.15.151:0/2589793317 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478628971 0 0) 0x7f932c0aba70 con 0x7f932c14a700 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9318003520 con 0x7f932c14a700 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912346337 0 0) 0x7f932c0aecb0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c0aba70 con 0x7f932c0ac5a0 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97616323 0 0) 0x7f932c0af150 con 0x7f932c0b0770 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c0aecb0 con 0x7f932c0b0770 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788343148 0 0) 0x7f932c0aba70 con 0x7f932c0ac5a0 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 >> v1:172.21.15.151:6790/0 conn(0x7f932c0b0770 legacy=0x7f932c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 >> v1:172.21.15.151:6789/0 conn(0x7f932c14a700 legacy=0x7f932c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:37.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.625+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c0ac9d0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f931c0019b0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: dump: 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.612 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f9331cff640 5 monclient: authenticate success, global_id 4525 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f9331cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f931c001c40 con 0x7f932c0ac5a0 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 1 -- 172.21.15.151:0/2589793317 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f931c001fd0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f932a7fc640 10 monclient: dump: 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.613 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f9331cff640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.626+0000 7f9331cff640 4 set_mon_vals no callback set 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 10 monclient: shutdown 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 1 -- 172.21.15.151:0/2589793317 >> v1:172.21.15.169:6789/0 conn(0x7f932c0ac5a0 legacy=0x7f932c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 1 -- 172.21.15.151:0/2589793317 shutdown_connections 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 1 -- 172.21.15.151:0/2589793317 >> 172.21.15.151:0/2589793317 conn(0x7f932c06b660 msgr2=0x7f932c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:37.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 1 -- 172.21.15.151:0/2589793317 shutdown_connections 2024-03-29T10:45:37.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.627+0000 7f9331cff640 1 -- 172.21.15.151:0/2589793317 wait complete. 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.628+0000 7f9331cff640 10 monclient: build_initial_monmap 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.628+0000 7f9331cff640 10 monclient: monmap: 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:37.622 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 librados: starting msgr at 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 librados: starting objecter 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 Processor -- start 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 -- start start 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 librados: setting wanted keys 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 1 librados: calling monclient init 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.629+0000 7f9331cff640 10 monclient: init 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:37.623 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-b con 0x7f932c0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-c con 0x7f932c0b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 10 monclient(hunting): picked mon.noname-a con 0x7f932c14a700 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c1db630 con 0x7f932c14a700 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f932b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f932c0ac5a0 0x7f932c128220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:38582/0 (socket says 172.21.15.151:38582) 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f932b7fe640 1 -- 172.21.15.151:0/1794263985 learned_addr learned my addr 172.21.15.151:0/1794263985 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.630+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c12a3a0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c147ca0 con 0x7f932c0b0770 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter ms_handle_connect 0x7f932c0ac5a0 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter ms_handle_connect 0x7f932c14a700 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9331cff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9331cff640 10 monclient(hunting): authenticate will time out at 1468.800811s 2024-03-29T10:45:37.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter ms_handle_connect 0x7f932c0b0770 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276538608 0 0) 0x7f932c1db630 con 0x7f932c14a700 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c004eb0 con 0x7f932c14a700 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096363786 0 0) 0x7f932c12a3a0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c1db630 con 0x7f932c0ac5a0 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699017832 0 0) 0x7f932c147ca0 con 0x7f932c0b0770 2024-03-29T10:45:37.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.631+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c005360 con 0x7f932c0b0770 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761161135 0 0) 0x7f930c004eb0 con 0x7f932c14a700 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f932c147ca0 con 0x7f932c14a700 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f931c0018b0 con 0x7f932c14a700 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): dump: 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.626 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2196954987 0 0) 0x7f932c1db630 con 0x7f932c0ac5a0 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f930c004eb0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f93240018c0 con 0x7f932c0ac5a0 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): dump: 2024-03-29T10:45:37.627 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114653765 0 0) 0x7f930c005360 con 0x7f932c0b0770 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f932c1db630 con 0x7f932c0b0770 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f9314001850 con 0x7f932c0b0770 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:37.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient(hunting): dump: 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3052363611 0 0) 0x7f932c147ca0 con 0x7f932c14a700 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 >> v1:172.21.15.151:6790/0 conn(0x7f932c0b0770 legacy=0x7f932c1dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 >> v1:172.21.15.169:6789/0 conn(0x7f932c0ac5a0 legacy=0x7f932c128220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:37.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 monclient: found mon.a 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c125760 con 0x7f932c14a700 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.632+0000 7f9323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 10 monclient: _renew_subs 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c1dc1d0 con 0x7f932c14a700 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 10 monclient: _renew_subs 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c129140 con 0x7f932c14a700 2024-03-29T10:45:37.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.633+0000 7f9331cff640 1 librados: init done 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f931c003c20 con 0x7f932c14a700 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f931c003f30 con 0x7f932c14a700 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9323fff640 10 monclient: dump: 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:37.631 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9329ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.634+0000 7f9331cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.635+0000 7f9331cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.635+0000 7f9331cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.635+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9310005080 con 0x7f932c14a700 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.638+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f931c019f00 con 0x7f932c14a700 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.638+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f931c083bf0 con 0x7f932c14a700 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.638+0000 7f9323fff640 10 client.4500.objecter ms_dispatch 0x7f932c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.638+0000 7f9323fff640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.638+0000 7f9323fff640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.639+0000 7f9323fff640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.639+0000 7f932b7fe640 10 monclient: get_auth_request con 0x7f930c06e2e0 auth_method 0 2024-03-29T10:45:37.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.639+0000 7f9323fff640 10 client.4500.objecter ms_handle_connect 0x7f930c06e2e0 2024-03-29T10:45:37.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.646+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f931c04f4d0 con 0x7f932c14a700 2024-03-29T10:45:37.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.646+0000 7f9323fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:37.646+0000 7f9323fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:38.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.019+0000 7f9331cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:38.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.019+0000 7f9331cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:38.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.019+0000 7f9331cff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:38.003 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.019+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9310005330 con 0x7f932c14a700 2024-03-29T10:45:38.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.021+0000 7f9323fff640 1 -- 172.21.15.151:0/1794263985 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f931c051400 con 0x7f932c14a700 2024-03-29T10:45:38.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.021+0000 7f9323fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:38.005 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.021+0000 7f9323fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:38.005 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:38.005 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:38.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 10 librados: watch_flush enter 2024-03-29T10:45:38.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 10 librados: watch_flush exit 2024-03-29T10:45:38.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:45:38.008 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 >> v1:172.21.15.151:6816/1549873000 conn(0x7f930c06e2e0 legacy=0x7f930c0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 10 monclient: shutdown 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 >> v1:172.21.15.151:6789/0 conn(0x7f932c14a700 legacy=0x7f932c147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 shutdown_connections 2024-03-29T10:45:38.009 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 >> 172.21.15.151:0/1794263985 conn(0x7f932c06b660 msgr2=0x7f932c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:38.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 shutdown_connections 2024-03-29T10:45:38.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 -- 172.21.15.151:0/1794263985 wait complete. 2024-03-29T10:45:38.010 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:38.025+0000 7f9331cff640 1 librados: shutdown 2024-03-29T10:45:38.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:44.831 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.984+0000 7fb2c1d5d640 10 monclient: get_monmap_and_config 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.984+0000 7fb2c1d5d640 10 monclient: build_initial_monmap 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.984+0000 7fb2c1d5d640 10 monclient: monmap: 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:44.986491+0000 2024-03-29T10:45:44.968 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:44.986491+0000 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:44.969 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.985+0000 7fb2c1d5d640 1 Processor -- start 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.985+0000 7fb2c1d5d640 1 -- start start 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.985+0000 7fb2c1d5d640 10 monclient: init 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2bc1457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc13f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc0a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:44.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc0aaef0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc145bd0 con 0x7fb2bc13f650 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc145eb0 con 0x7fb2bc1457f0 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.986+0000 7fb2c1d5d640 10 monclient(hunting): authenticate will time out at 1476.157402s 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.987+0000 7fb2bb7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb2bc1457f0 0x7fb2bc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:52776/0 (socket says 172.21.15.151:52776) 2024-03-29T10:45:44.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.987+0000 7fb2bb7fe640 1 -- 172.21.15.151:0/3655178713 learned_addr learned my addr 172.21.15.151:0/3655178713 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168451456 0 0) 0x7fb2bc145eb0 con 0x7fb2bc1457f0 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:45:44.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.989+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a4003520 con 0x7fb2bc1457f0 2024-03-29T10:45:44.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618975240 0 0) 0x7fb2a4003520 con 0x7fb2bc1457f0 2024-03-29T10:45:44.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 >> v1:172.21.15.169:6789/0 conn(0x7fb2bc13f650 legacy=0x7fb2bc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:44.976 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 >> v1:172.21.15.151:6789/0 conn(0x7fb2bc0a9d70 legacy=0x7fb2bc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 monclient: found mon.noname-b 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc146340 con 0x7fb2bc1457f0 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2c1d5d640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2c1d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb2ac001690 con 0x7fb2bc1457f0 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.990+0000 7fb2ba7fc640 10 monclient: dump: 2024-03-29T10:45:44.977 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2c1d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2ac0018f0 con 0x7fb2bc1457f0 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2c1d5d640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:44.978 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2c1d5d640 4 set_mon_vals no callback set 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 1 -- 172.21.15.151:0/3655178713 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb2ac001d40 con 0x7fb2bc1457f0 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.991+0000 7fb2ba7fc640 10 monclient: dump: 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:44.979 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.992+0000 7fb2c1d5d640 10 monclient: shutdown 2024-03-29T10:45:44.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.992+0000 7fb2c1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:44.980 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.992+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3655178713 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc1457f0 legacy=0x7fb2bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.995+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3655178713 shutdown_connections 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.995+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3655178713 >> 172.21.15.151:0/3655178713 conn(0x7fb2bc066a00 msgr2=0x7fb2bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.995+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3655178713 shutdown_connections 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.995+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3655178713 wait complete. 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.996+0000 7fb2c1d5d640 10 monclient: build_initial_monmap 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.996+0000 7fb2c1d5d640 10 monclient: monmap: 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:44.981 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.996+0000 7fb2c1d5d640 1 librados: starting msgr at 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.996+0000 7fb2c1d5d640 1 librados: starting objecter 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.996+0000 7fb2c1d5d640 1 Processor -- start 2024-03-29T10:45:44.982 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.997+0000 7fb2c1d5d640 1 -- start start 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.997+0000 7fb2c1d5d640 1 librados: setting wanted keys 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.997+0000 7fb2c1d5d640 1 librados: calling monclient init 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.997+0000 7fb2c1d5d640 10 monclient: init 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2bc0a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2bc13f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2bc1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc142380 con 0x7fb2bc13f650 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc141810 con 0x7fb2bc1457f0 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc141af0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:44.983 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:44.998+0000 7fb2c1d5d640 10 monclient(hunting): authenticate will time out at 1476.168786s 2024-03-29T10:45:44.985 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.000+0000 7fb2bb7fe640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 0x7fb2bc13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56132/0 (socket says 172.21.15.151:56132) 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.000+0000 7fb2bb7fe640 1 -- 172.21.15.151:0/3972793042 learned_addr learned my addr 172.21.15.151:0/3972793042 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.000+0000 7fb2a3fff640 10 client.?.objecter ms_handle_connect 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.000+0000 7fb2a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.000+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183507963 0 0) 0x7fb2bc141af0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8004eb0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530240807 0 0) 0x7fb2a8004eb0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2bc141af0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb2ac001650 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.986 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): dump: 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.987 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.001+0000 7fb2a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.002+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3192031231 0 0) 0x7fb2bc141af0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.002+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.169:6789/0 conn(0x7fb2bc1457f0 legacy=0x7fb2bc1e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6789/0 conn(0x7fb2bc13f650 legacy=0x7fb2bc142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 1 monclient: found mon.c 2024-03-29T10:45:44.990 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc13e030 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 10 monclient: _renew_subs 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc13e280 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 10 monclient: _renew_subs 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc144fa0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.991 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2c1d5d640 1 librados: init done 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2ac004a90 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb2ac004e60 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2a3fff640 10 monclient: dump: 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:44.992 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:44.993 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:44.993 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:44.993 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.003+0000 7fb2b9ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:44.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb2ac004fc0 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2c1d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:44.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fb2ac084d50 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2c1d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:44.994 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 10 client.4568.objecter ms_dispatch 0x7fb2bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2c1d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2a3fff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.007+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c005080 con 0x7fb2bc0a9d70 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.008+0000 7fb2baffd640 10 monclient: get_auth_request con 0x7fb2a806dd00 auth_method 0 2024-03-29T10:45:44.995 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.008+0000 7fb2a3fff640 10 client.4568.objecter ms_handle_connect 0x7fb2a806dd00 2024-03-29T10:45:45.000 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.017+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2ac050630 con 0x7fb2bc0a9d70 2024-03-29T10:45:45.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.017+0000 7fb2a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:45.002 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.017+0000 7fb2a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:45.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2c1d5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2c1d5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2c1d5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:45.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb29c005330 con 0x7fb2bc0a9d70 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2bb7fe640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 legacy=0x7fb2bc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2bb7fe640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 legacy=0x7fb2bc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2bb7fe640 1 --1- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 0x7fb2bc13fda0 :-1 s=OPENED pgs=867868 cs=1 l=1).handle_message read tag failed 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2bb7fe640 1 --1- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 0x7fb2bc13fda0 :-1 s=OPENED pgs=867868 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2a3fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.151:6790/0 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2bc0a9d70 legacy=0x7fb2bc13fda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:45:45.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2a3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.346+0000 7fb2a3fff640 10 monclient(hunting): picked mon.b con 0x7fb2bc1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 10 monclient(hunting): picked mon.c con 0x7fb2a8076790 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 10 monclient(hunting): picked mon.a con 0x7fb2bc13f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb2bc141af0 con 0x7fb2bc13f650 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb2bc142380 con 0x7fb2bc1457f0 2024-03-29T10:45:45.333 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb2bc141810 con 0x7fb2a8076790 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.347+0000 7fb2a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 10 client.4568.objecter ms_handle_connect 0x7fb2a8076790 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 10 client.4568.objecter resend_mon_ops 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479858884 0 0) 0x7fb2bc141810 con 0x7fb2a8076790 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb2a8080580 con 0x7fb2a8076790 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1793293344 0 0) 0x7fb2a8080580 con 0x7fb2a8076790 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.169:6789/0 conn(0x7fb2bc1457f0 legacy=0x7fb2a8074e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:45:45.334 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.349+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6789/0 conn(0x7fb2bc13f650 legacy=0x7fb2a807c930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 1 monclient: found mon.c 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fb2a8004eb0 con 0x7fb2a8076790 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2bc141af0 con 0x7fb2a8076790 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:45.337 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb2b4001860 con 0x7fb2a8076790 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.353+0000 7fb2a3fff640 10 monclient: dump: 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:45.338 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:45.341 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:45.341 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.355+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b40016b0 con 0x7fb2a8076790 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.355+0000 7fb2a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.355+0000 7fb2b9ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.356+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb2b4001ac0 con 0x7fb2a8076790 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.357+0000 7fb2a3fff640 1 -- 172.21.15.151:0/3972793042 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fb2b404f060 con 0x7fb2a8076790 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.357+0000 7fb2a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.342 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.357+0000 7fb2a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:45.344 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.360+0000 7fb2c1d5d640 10 librados: watch_flush enter 2024-03-29T10:45:45.344 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 10 librados: watch_flush exit 2024-03-29T10:45:45.344 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T10:45:45.344 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 10 client.4568.objecter successfully canceled tick 2024-03-29T10:45:45.345 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6816/1549873000 conn(0x7fb2a806dd00 legacy=0x7fb2a80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:45.345 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 10 monclient: shutdown 2024-03-29T10:45:45.346 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:45.346 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 >> v1:172.21.15.151:6790/0 conn(0x7fb2a8076790 legacy=0x7fb2a8078c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:45.346 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 shutdown_connections 2024-03-29T10:45:45.346 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.361+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 >> 172.21.15.151:0/3972793042 conn(0x7fb2bc066a00 msgr2=0x7fb2bc13b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:45.347 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.363+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 shutdown_connections 2024-03-29T10:45:45.347 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.363+0000 7fb2c1d5d640 1 -- 172.21.15.151:0/3972793042 wait complete. 2024-03-29T10:45:45.347 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:45.364+0000 7fb2c1d5d640 1 librados: shutdown 2024-03-29T10:45:45.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:52.173 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.319+0000 7fb230f82640 10 monclient: get_monmap_and_config 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.319+0000 7fb230f82640 10 monclient: build_initial_monmap 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.319+0000 7fb230f82640 10 monclient: monmap: 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:52.320582+0000 2024-03-29T10:45:52.302 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:52.320582+0000 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:52.303 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.320+0000 7fb230f82640 1 Processor -- start 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.320+0000 7fb230f82640 1 -- start start 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.320+0000 7fb230f82640 10 monclient: init 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:52.304 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-c con 0x7fb22c1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-a con 0x7fb22c13f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-b con 0x7fb22c0a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c0aaef0 con 0x7fb22c13f650 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c145bd0 con 0x7fb22c1457f0 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c145eb0 con 0x7fb22c0a9d70 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb230f82640 10 monclient(hunting): authenticate will time out at 1483.491498s 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.321+0000 7fb22a59c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb22c1457f0 0x7fb22c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:57124/0 (socket says 172.21.15.151:57124) 2024-03-29T10:45:52.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22a59c640 1 -- 172.21.15.151:0/3018657282 learned_addr learned my addr 172.21.15.151:0/3018657282 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:52.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879520194 0 0) 0x7fb22c145eb0 con 0x7fb22c0a9d70 2024-03-29T10:45:52.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:45:52.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.322+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb214003520 con 0x7fb22c0a9d70 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790653416 0 0) 0x7fb22c145bd0 con 0x7fb22c1457f0 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22c145eb0 con 0x7fb22c1457f0 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978067663 0 0) 0x7fb214003520 con 0x7fb22c0a9d70 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 >> v1:172.21.15.169:6789/0 conn(0x7fb22c1457f0 legacy=0x7fb22c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 >> v1:172.21.15.151:6789/0 conn(0x7fb22c13f650 legacy=0x7fb22c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 monclient: found mon.noname-b 2024-03-29T10:45:52.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c146340 con 0x7fb22c0a9d70 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: _finish_auth 0 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb224001990 con 0x7fb22c0a9d70 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: dump: 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.319 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb224001bb0 con 0x7fb22c0a9d70 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 1 -- 172.21.15.151:0/3018657282 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb224001f40 con 0x7fb22c0a9d70 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.324+0000 7fb22959a640 10 monclient: dump: 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.320 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.325+0000 7fb230f82640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.325+0000 7fb230f82640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.325+0000 7fb230f82640 4 set_mon_vals no callback set 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 10 monclient: shutdown 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 1 -- 172.21.15.151:0/3018657282 >> v1:172.21.15.151:6790/0 conn(0x7fb22c0a9d70 legacy=0x7fb22c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 1 -- 172.21.15.151:0/3018657282 shutdown_connections 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 1 -- 172.21.15.151:0/3018657282 >> 172.21.15.151:0/3018657282 conn(0x7fb22c066a00 msgr2=0x7fb22c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:52.321 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 1 -- 172.21.15.151:0/3018657282 shutdown_connections 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.326+0000 7fb230f82640 1 -- 172.21.15.151:0/3018657282 wait complete. 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.327+0000 7fb230f82640 10 monclient: build_initial_monmap 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.327+0000 7fb230f82640 10 monclient: monmap: 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.322 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.327+0000 7fb230f82640 1 librados: starting msgr at 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.327+0000 7fb230f82640 1 librados: starting objecter 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 1 Processor -- start 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 1 -- start start 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 1 librados: setting wanted keys 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 1 librados: calling monclient init 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 10 monclient: init 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.328+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-a con 0x7fb22c0a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-c con 0x7fb22c13f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb229d9b640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb22c13f650 0x7fb22c1db5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:41044/0 (socket says 172.21.15.151:41044) 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb229d9b640 1 -- 172.21.15.151:0/3925414229 learned_addr learned my addr 172.21.15.151:0/3925414229 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 10 monclient(hunting): picked mon.noname-b con 0x7fb22c1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:52.323 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c137870 con 0x7fb22c0a9d70 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 client.?.objecter ms_handle_connect 0x7fb22c13f650 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c141690 con 0x7fb22c1457f0 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 client.?.objecter ms_handle_connect 0x7fb22c1457f0 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c1dbcc0 con 0x7fb22c13f650 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb230f82640 10 monclient(hunting): authenticate will time out at 1483.499189s 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630907974 0 0) 0x7fb22c1dbcc0 con 0x7fb22c13f650 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:45:52.324 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.329+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb20c004eb0 con 0x7fb22c13f650 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2210008639 0 0) 0x7fb20c004eb0 con 0x7fb22c13f650 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22c1dbcc0 con 0x7fb22c13f650 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760025720 0 0) 0x7fb22c141690 con 0x7fb22c1457f0 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb20c004eb0 con 0x7fb22c1457f0 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb21c004080 con 0x7fb22c13f650 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): dump: 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:52.325 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717279679 0 0) 0x7fb20c004eb0 con 0x7fb22c1457f0 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22c141690 con 0x7fb22c1457f0 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 862959135 0 0) 0x7fb22c1dbcc0 con 0x7fb22c13f650 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.330+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 >> v1:172.21.15.169:6789/0 conn(0x7fb22c1457f0 legacy=0x7fb22c1e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.326 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 >> v1:172.21.15.151:6789/0 conn(0x7fb22c0a9d70 legacy=0x7fb22c144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 1 monclient: found mon.c 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c1dbe50 con 0x7fb22c13f650 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 monclient: _finish_auth 0 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 client.?.objecter ms_handle_connect 0x7fb22c0a9d70 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 10 monclient: _renew_subs 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.327 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c142a20 con 0x7fb22c13f650 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 10 monclient: _renew_subs 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c1420e0 con 0x7fb22c13f650 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.331+0000 7fb230f82640 1 librados: init done 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb21c004420 con 0x7fb22c13f650 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb21c001180 con 0x7fb22c13f650 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb21affd640 10 monclient: dump: 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:52.328 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb228d99640 4 set_mon_vals no callback set 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.332+0000 7fb230f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.333+0000 7fb230f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.333+0000 7fb230f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.333+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210005080 con 0x7fb22c13f650 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.333+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb21c0012e0 con 0x7fb22c13f650 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fb21c0849d0 con 0x7fb22c13f650 2024-03-29T10:45:52.329 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb21affd640 10 client.4574.objecter ms_dispatch 0x7fb22c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb21affd640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb21affd640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb22a59c640 10 monclient: get_auth_request con 0x7fb20c06e040 auth_method 0 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.334+0000 7fb21affd640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.335+0000 7fb21affd640 10 client.4574.objecter ms_handle_connect 0x7fb20c06e040 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.341+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb21c0502b0 con 0x7fb22c13f650 2024-03-29T10:45:52.330 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.342+0000 7fb21affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.331 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.342+0000 7fb21affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:52.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.645+0000 7fb230f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.645+0000 7fb230f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.627 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.645+0000 7fb230f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:45:52.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.645+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb210005330 con 0x7fb22c13f650 2024-03-29T10:45:52.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.646+0000 7fb21affd640 1 -- 172.21.15.151:0/3925414229 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fb21c0521e0 con 0x7fb22c13f650 2024-03-29T10:45:52.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.646+0000 7fb21affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.646+0000 7fb21affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:52.629 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:52.629 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:52.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 10 librados: watch_flush enter 2024-03-29T10:45:52.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 10 librados: watch_flush exit 2024-03-29T10:45:52.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T10:45:52.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 10 client.4574.objecter successfully canceled tick 2024-03-29T10:45:52.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 >> v1:172.21.15.151:6816/1549873000 conn(0x7fb20c06e040 legacy=0x7fb20c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 10 monclient: shutdown 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 >> v1:172.21.15.151:6790/0 conn(0x7fb22c13f650 legacy=0x7fb22c1db5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 shutdown_connections 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.649+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 >> 172.21.15.151:0/3925414229 conn(0x7fb22c066a00 msgr2=0x7fb22c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.650+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 shutdown_connections 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.650+0000 7fb230f82640 1 -- 172.21.15.151:0/3925414229 wait complete. 2024-03-29T10:45:52.633 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:52.650+0000 7fb230f82640 1 librados: shutdown 2024-03-29T10:45:52.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:59.456 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.626+0000 7fca434ae640 10 monclient: get_monmap_and_config 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.626+0000 7fca434ae640 10 monclient: build_initial_monmap 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.626+0000 7fca434ae640 10 monclient: monmap: 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:45:59.627314+0000 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:45:59.627314+0000 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:59.609 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.627+0000 7fca434ae640 1 Processor -- start 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.627+0000 7fca434ae640 1 -- start start 2024-03-29T10:45:59.610 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.627+0000 7fca434ae640 10 monclient: init 2024-03-29T10:45:59.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.628+0000 7fca434ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:59.611 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.629+0000 7fca434ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:59.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-a con 0x7fca34036a90 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:59.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-c con 0x7fca340352e0 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:59.614 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-b con 0x7fca3402ccf0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3402d3d0 con 0x7fca34036a90 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3402e370 con 0x7fca340352e0 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340c30c0 con 0x7fca3402ccf0 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.630+0000 7fca434ae640 10 monclient(hunting): authenticate will time out at 1490.800299s 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca42cad640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fca3402ccf0 0x7fca340c5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:58320/0 (socket says 172.21.15.151:58320) 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca42cad640 1 -- 172.21.15.151:0/1041271352 learned_addr learned my addr 172.21.15.151:0/1041271352 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713447052 0 0) 0x7fca340c30c0 con 0x7fca3402ccf0 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:45:59.615 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.631+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca28003520 con 0x7fca3402ccf0 2024-03-29T10:45:59.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.632+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2949449845 0 0) 0x7fca28003520 con 0x7fca3402ccf0 2024-03-29T10:45:59.616 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.632+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 >> v1:172.21.15.169:6789/0 conn(0x7fca340352e0 legacy=0x7fca340c5790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 >> v1:172.21.15.151:6789/0 conn(0x7fca34036a90 legacy=0x7fca340c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 1 monclient: found mon.noname-b 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.633+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340c3250 con 0x7fca3402ccf0 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: _finish_auth 0 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fca3c04f730 con 0x7fca3402ccf0 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.619 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: dump: 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca434ae640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca434ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:59.620 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca3c075f10 con 0x7fca3402ccf0 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 1 -- 172.21.15.151:0/1041271352 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fca3c0758b0 con 0x7fca3402ccf0 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca414aa640 10 monclient: dump: 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:59.621 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca434ae640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.634+0000 7fca434ae640 4 set_mon_vals no callback set 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.635+0000 7fca434ae640 10 monclient: shutdown 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.635+0000 7fca434ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.635+0000 7fca434ae640 1 -- 172.21.15.151:0/1041271352 >> v1:172.21.15.151:6790/0 conn(0x7fca3402ccf0 legacy=0x7fca340c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.636+0000 7fca434ae640 1 -- 172.21.15.151:0/1041271352 shutdown_connections 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.636+0000 7fca434ae640 1 -- 172.21.15.151:0/1041271352 >> 172.21.15.151:0/1041271352 conn(0x7fca340058f0 msgr2=0x7fca34005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.636+0000 7fca434ae640 1 -- 172.21.15.151:0/1041271352 shutdown_connections 2024-03-29T10:45:59.622 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.636+0000 7fca434ae640 1 -- 172.21.15.151:0/1041271352 wait complete. 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.637+0000 7fca434ae640 10 monclient: build_initial_monmap 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.637+0000 7fca434ae640 10 monclient: monmap: 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.624 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.637+0000 7fca434ae640 1 librados: starting msgr at 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 1 librados: starting objecter 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 1 Processor -- start 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 1 -- start start 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 1 librados: setting wanted keys 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 1 librados: calling monclient init 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.638+0000 7fca434ae640 10 monclient: init 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.639+0000 7fca434ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.639+0000 7fca434ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:59.625 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-a con 0x7fca3402ccf0 addr v1:172.21.15.151:6789/0 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-c con 0x7fca340352e0 addr v1:172.21.15.151:6790/0 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 10 monclient(hunting): picked mon.noname-b con 0x7fca34036a90 addr v1:172.21.15.169:6789/0 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340c8a40 con 0x7fca3402ccf0 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340c6a50 con 0x7fca34036a90 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca340bfd90 con 0x7fca340352e0 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:59.628 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.640+0000 7fca434ae640 10 monclient(hunting): authenticate will time out at 1490.809919s 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca424ac640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fca3402ccf0 0x7fca340bc330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59467/0 (socket says 172.21.15.151:59467) 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca424ac640 1 -- 172.21.15.151:0/2477341203 learned_addr learned my addr 172.21.15.151:0/2477341203 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 client.?.objecter ms_handle_connect 0x7fca3402ccf0 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87980515 0 0) 0x7fca340c8a40 con 0x7fca3402ccf0 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.643+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca20004eb0 con 0x7fca3402ccf0 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 client.?.objecter ms_handle_connect 0x7fca340352e0 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.629 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 client.?.objecter ms_handle_connect 0x7fca34036a90 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108103195 0 0) 0x7fca20004eb0 con 0x7fca3402ccf0 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca340c8a40 con 0x7fca3402ccf0 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603095480 0 0) 0x7fca340bfd90 con 0x7fca340352e0 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca20004eb0 con 0x7fca340352e0 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fca38001890 con 0x7fca3402ccf0 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:59.630 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586410866 0 0) 0x7fca340c6a50 con 0x7fca34036a90 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.631 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca340bfd90 con 0x7fca34036a90 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468529739 0 0) 0x7fca340c8a40 con 0x7fca3402ccf0 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.644+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 >> v1:172.21.15.151:6790/0 conn(0x7fca340352e0 legacy=0x7fca3415a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 >> v1:172.21.15.169:6789/0 conn(0x7fca34036a90 legacy=0x7fca3415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 1 monclient: found mon.a 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340bff20 con 0x7fca3402ccf0 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:59.632 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.645+0000 7fca32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:59.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 5 monclient: authenticate success, global_id 4506 2024-03-29T10:45:59.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 10 monclient: _renew_subs 2024-03-29T10:45:59.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:59.634 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca340bca40 con 0x7fca3402ccf0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 10 monclient: _renew_subs 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca340c7400 con 0x7fca3402ccf0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.646+0000 7fca434ae640 1 librados: init done 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca30ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca30ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca34009d90 con 0x7fca3402ccf0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca38001a60 con 0x7fca3402ccf0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fca38003830 con 0x7fca3402ccf0 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.635 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca32ffd640 10 monclient: dump: 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.650+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fca38003a50 con 0x7fca3402ccf0 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.649+0000 7fca40ca9640 4 set_mon_vals no callback set 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.651+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fca380849f0 con 0x7fca3402ccf0 2024-03-29T10:45:59.636 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.651+0000 7fca32ffd640 10 client.4506.objecter ms_dispatch 0x7fca3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:59.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.651+0000 7fca32ffd640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:59.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.651+0000 7fca32ffd640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:59.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.651+0000 7fca32ffd640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T10:45:59.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.652+0000 7fca41cab640 10 monclient: get_auth_request con 0x7fca2006da20 auth_method 0 2024-03-29T10:45:59.637 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.653+0000 7fca32ffd640 10 client.4506.objecter ms_handle_connect 0x7fca2006da20 2024-03-29T10:45:59.642 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.658+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca380502d0 con 0x7fca3402ccf0 2024-03-29T10:45:59.642 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.658+0000 7fca32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.643 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.658+0000 7fca32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:59.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca30ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca30ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:45:59.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca34004cd0 con 0x7fca3402ccf0 2024-03-29T10:45:59.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca32ffd640 1 -- 172.21.15.151:0/2477341203 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fca38052200 con 0x7fca3402ccf0 2024-03-29T10:45:59.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.981+0000 7fca32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:59.965 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:45:59.965 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:59.968 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.985+0000 7fca30ff9640 10 librados: watch_flush enter 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 10 librados: watch_flush exit 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 10 client.4506.objecter successfully canceled tick 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 >> v1:172.21.15.151:6816/1549873000 conn(0x7fca2006da20 legacy=0x7fca2006fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 10 monclient: shutdown 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 >> v1:172.21.15.151:6789/0 conn(0x7fca3402ccf0 legacy=0x7fca340bc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 shutdown_connections 2024-03-29T10:45:59.969 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.986+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 >> 172.21.15.151:0/2477341203 conn(0x7fca340058f0 msgr2=0x7fca3402afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:59.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.987+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 shutdown_connections 2024-03-29T10:45:59.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.989+0000 7fca30ff9640 1 -- 172.21.15.151:0/2477341203 wait complete. 2024-03-29T10:45:59.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:45:59.990+0000 7fca30ff9640 1 librados: shutdown 2024-03-29T10:45:59.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:06.798 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:06.950 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.967+0000 7f8a6836b640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.967+0000 7f8a6836b640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.967+0000 7f8a6836b640 10 monclient: monmap: 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:06.968178+0000 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:06.968178+0000 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.951 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.968+0000 7f8a6836b640 1 Processor -- start 2024-03-29T10:46:06.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.968+0000 7f8a6836b640 1 -- start start 2024-03-29T10:46:06.952 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.968+0000 7f8a6836b640 10 monclient: init 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.969+0000 7f8a6836b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.969+0000 7f8a6836b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.969+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a600a9f50 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.969+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a600aa330 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a60138390 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:06.953 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a600ab6c0 con 0x7f8a600a9f50 2024-03-29T10:46:06.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6013eb20 con 0x7f8a600aa330 2024-03-29T10:46:06.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a600b4310 con 0x7f8a60138390 2024-03-29T10:46:06.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.954 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.970+0000 7f8a6836b640 10 monclient(hunting): authenticate will time out at 1498.140394s 2024-03-29T10:46:06.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.971+0000 7f8a66908640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f8a60138390 0x7f8a60149f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:39414/0 (socket says 172.21.15.151:39414) 2024-03-29T10:46:06.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.971+0000 7f8a66908640 1 -- 172.21.15.151:0/1671487561 learned_addr learned my addr 172.21.15.151:0/1671487561 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:06.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172154874 0 0) 0x7f8a600b4310 con 0x7f8a60138390 2024-03-29T10:46:06.956 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a4c005730 con 0x7f8a60138390 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801511820 0 0) 0x7f8a4c005730 con 0x7f8a60138390 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 >> v1:172.21.15.169:6789/0 conn(0x7f8a600aa330 legacy=0x7f8a600adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 >> v1:172.21.15.151:6789/0 conn(0x7f8a600a9f50 legacy=0x7f8a6013f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.972+0000 7f8a65105640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 1 monclient: found mon.noname-b 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6013c1a0 con 0x7f8a60138390 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.957 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8a58001a10 con 0x7f8a60138390 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: dump: 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:06.958 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a6836b640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a6836b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a58001ca0 con 0x7f8a60138390 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 1 -- 172.21.15.151:0/1671487561 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8a58002070 con 0x7f8a60138390 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a65105640 10 monclient: dump: 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.959 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a6836b640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.973+0000 7f8a6836b640 4 set_mon_vals no callback set 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.974+0000 7f8a6836b640 10 monclient: shutdown 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.974+0000 7f8a6836b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.960 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.974+0000 7f8a6836b640 1 -- 172.21.15.151:0/1671487561 >> v1:172.21.15.151:6790/0 conn(0x7f8a60138390 legacy=0x7f8a60149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.961 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.974+0000 7f8a6836b640 1 -- 172.21.15.151:0/1671487561 shutdown_connections 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.974+0000 7f8a6836b640 1 -- 172.21.15.151:0/1671487561 >> 172.21.15.151:0/1671487561 conn(0x7f8a600b35e0 msgr2=0x7f8a600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.975+0000 7f8a6836b640 1 -- 172.21.15.151:0/1671487561 shutdown_connections 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.975+0000 7f8a6836b640 1 -- 172.21.15.151:0/1671487561 wait complete. 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.976+0000 7f8a6836b640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.976+0000 7f8a6836b640 10 monclient: monmap: 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.962 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.976+0000 7f8a6836b640 1 librados: starting msgr at 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.976+0000 7f8a6836b640 1 librados: starting objecter 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.976+0000 7f8a6836b640 1 Processor -- start 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 1 -- start start 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 1 librados: setting wanted keys 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 1 librados: calling monclient init 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 10 monclient: init 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.963 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:06.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a600a9f50 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:06.964 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.977+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a60138390 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.979+0000 7f8a6836b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a600702c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.979+0000 7f8a6836b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a60071ca0 con 0x7f8a60138390 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.979+0000 7f8a6836b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a600716e0 con 0x7f8a600702c0 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.979+0000 7f8a6836b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a60138040 con 0x7f8a600a9f50 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.980+0000 7f8a6836b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.980+0000 7f8a6836b640 10 monclient(hunting): authenticate will time out at 1498.149604s 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.981+0000 7f8a65906640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8a60138390 0x7f8a6006fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:35629/0 (socket says 172.21.15.151:35629) 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.981+0000 7f8a65906640 1 -- 172.21.15.151:0/3260614698 learned_addr learned my addr 172.21.15.151:0/3260614698 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 client.?.objecter ms_handle_connect 0x7f8a60138390 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.970 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 client.?.objecter ms_handle_connect 0x7f8a600a9f50 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380546598 0 0) 0x7f8a60071ca0 con 0x7f8a60138390 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.982+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a44004eb0 con 0x7f8a60138390 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181247308 0 0) 0x7f8a60138040 con 0x7f8a600a9f50 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a60071ca0 con 0x7f8a600a9f50 2024-03-29T10:46:06.971 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553680774 0 0) 0x7f8a44004eb0 con 0x7f8a60138390 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a60138040 con 0x7f8a60138390 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8a50001960 con 0x7f8a60138390 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:06.972 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.983+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123597828 0 0) 0x7f8a60071ca0 con 0x7f8a600a9f50 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a44004eb0 con 0x7f8a600a9f50 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1718008008 0 0) 0x7f8a60138040 con 0x7f8a60138390 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 >> v1:172.21.15.151:6790/0 conn(0x7f8a600a9f50 legacy=0x7f8a60072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 >> v1:172.21.15.169:6789/0 conn(0x7f8a600702c0 legacy=0x7f8a601e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 monclient: found mon.a 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6006f860 con 0x7f8a60138390 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.973 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a500035f0 con 0x7f8a60138390 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 10 monclient: _renew_subs 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6006f560 con 0x7f8a60138390 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 10 monclient: _renew_subs 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a601e7800 con 0x7f8a60138390 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.984+0000 7f8a6836b640 1 librados: init done 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8a50003940 con 0x7f8a60138390 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a57fff640 4 set_mon_vals no callback set 2024-03-29T10:46:06.974 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.985+0000 7f8a56ffd640 10 monclient: dump: 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:06.975 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.987+0000 7f8a6836b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.987+0000 7f8a6836b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.987+0000 7f8a6836b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.987+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a48005080 con 0x7f8a60138390 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.987+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f8a50003aa0 con 0x7f8a60138390 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f8a50084a00 con 0x7f8a60138390 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a56ffd640 10 client.4512.objecter ms_dispatch 0x7f8a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a56ffd640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a56ffd640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a56ffd640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.988+0000 7f8a66107640 10 monclient: get_auth_request con 0x7f8a4406de20 auth_method 0 2024-03-29T10:46:06.977 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.989+0000 7f8a56ffd640 10 client.4512.objecter ms_handle_connect 0x7f8a4406de20 2024-03-29T10:46:06.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.996+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a500502e0 con 0x7f8a60138390 2024-03-29T10:46:06.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.996+0000 7f8a56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.979 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:06.996+0000 7f8a56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:07.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.323+0000 7f8a6836b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.323+0000 7f8a6836b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.323+0000 7f8a6836b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:07.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.323+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a48005600 con 0x7f8a60138390 2024-03-29T10:46:07.307 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:07.307 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:07.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.324+0000 7f8a56ffd640 1 -- 172.21.15.151:0/3260614698 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f8a50052210 con 0x7f8a60138390 2024-03-29T10:46:07.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.324+0000 7f8a56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.324+0000 7f8a56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:07.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 10 librados: watch_flush enter 2024-03-29T10:46:07.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 10 librados: watch_flush exit 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 10 client.4512.objecter successfully canceled tick 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 >> v1:172.21.15.151:6816/1549873000 conn(0x7f8a4406de20 legacy=0x7f8a440702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 10 monclient: shutdown 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:07.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.327+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 >> v1:172.21.15.151:6789/0 conn(0x7f8a60138390 legacy=0x7f8a6006fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.328+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 shutdown_connections 2024-03-29T10:46:07.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.328+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 >> 172.21.15.151:0/3260614698 conn(0x7f8a600b35e0 msgr2=0x7f8a600b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:07.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.329+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 shutdown_connections 2024-03-29T10:46:07.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.329+0000 7f8a6836b640 1 -- 172.21.15.151:0/3260614698 wait complete. 2024-03-29T10:46:07.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:07.329+0000 7f8a6836b640 1 librados: shutdown 2024-03-29T10:46:07.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:14.136 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:14.286 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.304+0000 7f4bcd3cd640 10 monclient: get_monmap_and_config 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.304+0000 7f4bcd3cd640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.304+0000 7f4bcd3cd640 10 monclient: monmap: 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:14.305151+0000 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:14.305151+0000 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:14.287 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:14.288 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.305+0000 7f4bcd3cd640 1 Processor -- start 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.305+0000 7f4bcd3cd640 1 -- start start 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.305+0000 7f4bcd3cd640 10 monclient: init 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc81457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:14.290 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc813f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc80a9d70 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80aaef0 con 0x7f4bc813f650 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8145bd0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8145eb0 con 0x7f4bc81457f0 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bcd3cd640 10 monclient(hunting): authenticate will time out at 1505.475709s 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bc77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4bc80a9d70 0x7f4bc814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:49934/0 (socket says 172.21.15.151:49934) 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bc77fe640 1 -- 172.21.15.151:0/1552285351 learned_addr learned my addr 172.21.15.151:0/1552285351 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234763650 0 0) 0x7f4bc8145bd0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bc5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.306+0000 7f4bc5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.291 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb0003520 con 0x7f4bc80a9d70 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667095820 0 0) 0x7f4bb0003520 con 0x7f4bc80a9d70 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 >> v1:172.21.15.151:6790/0 conn(0x7f4bc81457f0 legacy=0x7f4bc813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 >> v1:172.21.15.151:6789/0 conn(0x7f4bc813f650 legacy=0x7f4bc80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 monclient: found mon.noname-c 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8146340 con 0x7f4bc80a9d70 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.292 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4bbc001a10 con 0x7f4bc80a9d70 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bc5ffb640 10 monclient: dump: 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.293 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bcd3cd640 5 monclient: authenticate success, global_id 4540 2024-03-29T10:46:14.294 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.307+0000 7f4bcd3cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:14.305 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bbc001c30 con 0x7f4bc80a9d70 2024-03-29T10:46:14.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bcd3cd640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:14.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bcd3cd640 4 set_mon_vals no callback set 2024-03-29T10:46:14.306 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 1 -- 172.21.15.151:0/1552285351 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4bbc001fc0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bc5ffb640 10 monclient: dump: 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.307 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bcd3cd640 10 monclient: shutdown 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bcd3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.308+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/1552285351 >> v1:172.21.15.169:6789/0 conn(0x7f4bc80a9d70 legacy=0x7f4bc814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.309+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/1552285351 shutdown_connections 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.309+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/1552285351 >> 172.21.15.151:0/1552285351 conn(0x7f4bc8066a00 msgr2=0x7f4bc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:14.308 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.309+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/1552285351 shutdown_connections 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.309+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/1552285351 wait complete. 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.310+0000 7f4bcd3cd640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.310+0000 7f4bcd3cd640 10 monclient: monmap: 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:14.309 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.310+0000 7f4bcd3cd640 1 librados: starting msgr at 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.310+0000 7f4bcd3cd640 1 librados: starting objecter 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.311+0000 7f4bcd3cd640 1 Processor -- start 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.311+0000 7f4bcd3cd640 1 -- start start 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.311+0000 7f4bcd3cd640 1 librados: setting wanted keys 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.311+0000 7f4bcd3cd640 1 librados: calling monclient init 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.311+0000 7f4bcd3cd640 10 monclient: init 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc80a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:14.310 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc81457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80ac3b0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc813b4e0 con 0x7f4bc813f650 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc813a860 con 0x7f4bc81457f0 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bcd3cd640 10 monclient(hunting): authenticate will time out at 1505.481751s 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bc6ffd640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4bc80a9d70 0x7f4bc813ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:45588/0 (socket says 172.21.15.151:45588) 2024-03-29T10:46:14.311 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4bc6ffd640 1 -- 172.21.15.151:0/4146889730 learned_addr learned my addr 172.21.15.151:0/4146889730 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4baf7fe640 10 client.?.objecter ms_handle_connect 0x7f4bc80a9d70 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747550504 0 0) 0x7f4bc80ac3b0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.312+0000 7f4baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4004eb0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 client.?.objecter ms_handle_connect 0x7f4bc81457f0 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158705341 0 0) 0x7f4bc813a860 con 0x7f4bc81457f0 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.312 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc80ac3b0 con 0x7f4bc81457f0 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077316714 0 0) 0x7f4bb4004eb0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc813a860 con 0x7f4bc80a9d70 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4bc0001890 con 0x7f4bc80a9d70 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.313 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 client.?.objecter ms_handle_connect 0x7f4bc813f650 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.313+0000 7f4baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737345971 0 0) 0x7f4bc813b4e0 con 0x7f4bc813f650 2024-03-29T10:46:14.314 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4004eb0 con 0x7f4bc813f650 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3488585605 0 0) 0x7f4bc80ac3b0 con 0x7f4bc81457f0 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc813b4e0 con 0x7f4bc81457f0 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4bbc001da0 con 0x7f4bc81457f0 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.315 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1724258344 0 0) 0x7f4bc813a860 con 0x7f4bc80a9d70 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 >> v1:172.21.15.151:6790/0 conn(0x7f4bc81457f0 legacy=0x7f4bc81d6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 >> v1:172.21.15.169:6789/0 conn(0x7f4bc813f650 legacy=0x7f4bc8136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 monclient: found mon.a 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:14.316 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8137700 con 0x7f4bc80a9d70 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4baf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4bcd3cd640 5 monclient: authenticate success, global_id 4518 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.314+0000 7f4bcd3cd640 10 monclient: _renew_subs 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc8137950 con 0x7f4bc80a9d70 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 10 monclient: _renew_subs 2024-03-29T10:46:14.317 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc8142370 con 0x7f4bc80a9d70 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bcd3cd640 1 librados: init done 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bc0003610 con 0x7f4bc80a9d70 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4baf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4bc0001d80 con 0x7f4bc80a9d70 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4baf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.315+0000 7f4bc57fa640 4 set_mon_vals no callback set 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4baf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4bcd3cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4baf7fe640 10 monclient: dump: 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.318 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4bcd3cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4bcd3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.316+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8005080 con 0x7f4bc80a9d70 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.318+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4bc0003960 con 0x7f4bc80a9d70 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f4bc0084830 con 0x7f4bc80a9d70 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4bc67fc640 10 monclient: get_auth_request con 0x7f4bb406df70 auth_method 0 2024-03-29T10:46:14.319 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4baf7fe640 10 client.4518.objecter ms_dispatch 0x7f4bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4baf7fe640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4baf7fe640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.319+0000 7f4baf7fe640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.320+0000 7f4baf7fe640 10 client.4518.objecter ms_handle_connect 0x7f4bb406df70 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.328+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bc0050170 con 0x7f4bc80a9d70 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.328+0000 7f4baf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.320 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.328+0000 7f4baf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:14.690 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.706+0000 7f4bcd3cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.706+0000 7f4bcd3cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.706+0000 7f4bcd3cd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.706+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba8005330 con 0x7f4bc80a9d70 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.707+0000 7f4baf7fe640 1 -- 172.21.15.151:0/4146889730 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f4bc00520a0 con 0x7f4bc80a9d70 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.707+0000 7f4baf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.707+0000 7f4baf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:14.692 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:14.692 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:14.695 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 10 librados: watch_flush enter 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 10 librados: watch_flush exit 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 10 client.4518.objecter successfully canceled tick 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4bb406df70 legacy=0x7f4bb4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 10 monclient: shutdown 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:14.696 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.711+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 >> v1:172.21.15.151:6789/0 conn(0x7f4bc80a9d70 legacy=0x7f4bc813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.712+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 shutdown_connections 2024-03-29T10:46:14.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.712+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 >> 172.21.15.151:0/4146889730 conn(0x7f4bc8066a00 msgr2=0x7f4bc80b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:14.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.712+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 shutdown_connections 2024-03-29T10:46:14.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.712+0000 7f4bcd3cd640 1 -- 172.21.15.151:0/4146889730 wait complete. 2024-03-29T10:46:14.697 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:14.712+0000 7f4bcd3cd640 1 librados: shutdown 2024-03-29T10:46:14.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:21.516 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:21.672 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.690+0000 7f26825c6640 10 monclient: get_monmap_and_config 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.690+0000 7f26825c6640 10 monclient: build_initial_monmap 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.690+0000 7f26825c6640 10 monclient: monmap: 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:21.691140+0000 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:21.691140+0000 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:21.673 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:21.674 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:21.674 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:21.674 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:21.676 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.693+0000 7f26825c6640 1 Processor -- start 2024-03-29T10:46:21.678 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.695+0000 7f26825c6640 1 -- start start 2024-03-29T10:46:21.678 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.695+0000 7f26825c6640 10 monclient: init 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-a con 0x7f267c0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-b con 0x7f267c14a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-c con 0x7f267c0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c0aba70 con 0x7f267c0b0770 2024-03-29T10:46:21.679 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c0aecb0 con 0x7f267c0ac5a0 2024-03-29T10:46:21.680 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c0af150 con 0x7f267c14a700 2024-03-29T10:46:21.680 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:21.680 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.696+0000 7f26825c6640 10 monclient(hunting): authenticate will time out at 1512.865730s 2024-03-29T10:46:21.680 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267bfff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f267c0b0770 0x7f267c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:53984/0 (socket says 172.21.15.151:53984) 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267bfff640 1 -- 172.21.15.151:0/506567453 learned_addr learned my addr 172.21.15.151:0/506567453 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823832891 0 0) 0x7f267c0aba70 con 0x7f267c0b0770 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.698+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2668003520 con 0x7f267c0b0770 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.699+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3750877832 0 0) 0x7f2668003520 con 0x7f267c0b0770 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.699+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 >> v1:172.21.15.151:6790/0 conn(0x7f267c14a700 legacy=0x7f267c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:21.682 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.699+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 >> v1:172.21.15.169:6789/0 conn(0x7f267c0ac5a0 legacy=0x7f267c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:21.686 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:21.686 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 1 monclient: found mon.noname-a 2024-03-29T10:46:21.686 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:21.686 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c0ac9d0 con 0x7f267c0b0770 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: _finish_auth 0 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f26825c6640 5 monclient: authenticate success, global_id 4521 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f26825c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f266c001a10 con 0x7f267c0b0770 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f267affd640 10 monclient: dump: 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:21.687 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.703+0000 7f26825c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.704+0000 7f267affd640 1 -- 172.21.15.151:0/506567453 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f266c001d20 con 0x7f267c0b0770 2024-03-29T10:46:21.688 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.704+0000 7f267affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.704+0000 7f26825c6640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.704+0000 7f26825c6640 4 set_mon_vals no callback set 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.705+0000 7f26825c6640 10 monclient: shutdown 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.705+0000 7f26825c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.705+0000 7f26825c6640 1 -- 172.21.15.151:0/506567453 >> v1:172.21.15.151:6789/0 conn(0x7f267c0b0770 legacy=0x7f267c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.706+0000 7f26825c6640 1 -- 172.21.15.151:0/506567453 shutdown_connections 2024-03-29T10:46:21.689 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.706+0000 7f26825c6640 1 -- 172.21.15.151:0/506567453 >> 172.21.15.151:0/506567453 conn(0x7f267c06b660 msgr2=0x7f267c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:21.690 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.706+0000 7f26825c6640 1 -- 172.21.15.151:0/506567453 shutdown_connections 2024-03-29T10:46:21.690 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.708+0000 7f26825c6640 1 -- 172.21.15.151:0/506567453 wait complete. 2024-03-29T10:46:21.691 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.709+0000 7f26825c6640 10 monclient: build_initial_monmap 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.709+0000 7f26825c6640 10 monclient: monmap: 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:21.692 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:21.693 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.709+0000 7f26825c6640 1 librados: starting msgr at 2024-03-29T10:46:21.693 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.709+0000 7f26825c6640 1 librados: starting objecter 2024-03-29T10:46:21.693 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.710+0000 7f26825c6640 1 Processor -- start 2024-03-29T10:46:21.703 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.710+0000 7f26825c6640 1 -- start start 2024-03-29T10:46:21.703 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.710+0000 7f26825c6640 1 librados: setting wanted keys 2024-03-29T10:46:21.703 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.710+0000 7f26825c6640 1 librados: calling monclient init 2024-03-29T10:46:21.703 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.710+0000 7f26825c6640 10 monclient: init 2024-03-29T10:46:21.703 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-b con 0x7f267c0ac5a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-a con 0x7f267c0b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient(hunting): picked mon.noname-c con 0x7f267c14a700 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c1dc1f0 con 0x7f267c0b0770 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c1dbcd0 con 0x7f267c0ac5a0 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c1db9f0 con 0x7f267c14a700 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f26825c6640 10 monclient(hunting): authenticate will time out at 1512.880788s 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f267b7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f267c0b0770 0x7f267c1474b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:57568/0 (socket says 172.21.15.151:57568) 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.711+0000 7f267b7fe640 1 -- 172.21.15.151:0/1631133537 learned_addr learned my addr 172.21.15.151:0/1631133537 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 client.?.objecter ms_handle_connect 0x7f267c0b0770 2024-03-29T10:46:21.704 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032502561 0 0) 0x7f267c1dc1f0 con 0x7f267c0b0770 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.712+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c004eb0 con 0x7f267c0b0770 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2397451481 0 0) 0x7f265c004eb0 con 0x7f267c0b0770 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f267c1dc1f0 con 0x7f267c0b0770 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f26740017d0 con 0x7f267c0b0770 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:21.705 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3846540887 0 0) 0x7f267c1dc1f0 con 0x7f267c0b0770 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 >> v1:172.21.15.151:6790/0 conn(0x7f267c14a700 legacy=0x7f267c1239d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.713+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 >> v1:172.21.15.169:6789/0 conn(0x7f267c0ac5a0 legacy=0x7f267c1db2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 1 monclient: found mon.a 2024-03-29T10:46:21.706 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c1481e0 con 0x7f267c0b0770 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.718+0000 7f2678ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 5 monclient: authenticate success, global_id 4524 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 10 monclient: _renew_subs 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c147bc0 con 0x7f267c0b0770 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 10 monclient: _renew_subs 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c0702e0 con 0x7f267c0b0770 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f26825c6640 1 librados: init done 2024-03-29T10:46:21.707 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2674001930 con 0x7f267c0b0770 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2674004a40 con 0x7f267c0b0770 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2678ff9640 10 monclient: dump: 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:21.708 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.719+0000 7f2679ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.720+0000 7f26825c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.720+0000 7f26825c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.720+0000 7f26825c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.720+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660005080 con 0x7f267c0b0770 2024-03-29T10:46:21.709 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f267401aa10 con 0x7f267c0b0770 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f267bfff640 10 monclient: get_auth_request con 0x7f265c06dc90 auth_method 0 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f2674084a80 con 0x7f267c0b0770 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 10 client.4524.objecter ms_dispatch 0x7f267c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.725+0000 7f2678ff9640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T10:46:21.710 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.726+0000 7f2678ff9640 10 client.4524.objecter ms_handle_connect 0x7f265c06dc90 2024-03-29T10:46:21.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.735+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2674050190 con 0x7f267c0b0770 2024-03-29T10:46:21.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.735+0000 7f2678ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.718 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:21.735+0000 7f2678ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:22.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.094+0000 7f26825c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.094+0000 7f26825c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.094+0000 7f26825c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:22.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.094+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2660005330 con 0x7f267c0b0770 2024-03-29T10:46:22.080 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:22.081 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:22.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.095+0000 7f2678ff9640 1 -- 172.21.15.151:0/1631133537 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f2674052290 con 0x7f267c0b0770 2024-03-29T10:46:22.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.095+0000 7f2678ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:22.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.095+0000 7f2678ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:22.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.099+0000 7f26825c6640 10 librados: watch_flush enter 2024-03-29T10:46:22.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 10 librados: watch_flush exit 2024-03-29T10:46:22.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T10:46:22.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 10 client.4524.objecter successfully canceled tick 2024-03-29T10:46:22.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 >> v1:172.21.15.151:6816/1549873000 conn(0x7f265c06dc90 legacy=0x7f265c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 10 monclient: shutdown 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 >> v1:172.21.15.151:6789/0 conn(0x7f267c0b0770 legacy=0x7f267c1474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 shutdown_connections 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.101+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 >> 172.21.15.151:0/1631133537 conn(0x7f267c06b660 msgr2=0x7f267c149280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.102+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 shutdown_connections 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.102+0000 7f26825c6640 1 -- 172.21.15.151:0/1631133537 wait complete. 2024-03-29T10:46:22.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:22.103+0000 7f26825c6640 1 librados: shutdown 2024-03-29T10:46:22.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:28.914 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:29.057 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.075+0000 7fe82d6a1640 10 monclient: get_monmap_and_config 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.075+0000 7fe82d6a1640 10 monclient: build_initial_monmap 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.075+0000 7fe82d6a1640 10 monclient: monmap: 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:29.076206+0000 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:29.076206+0000 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:29.058 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:29.059 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.076+0000 7fe82d6a1640 1 Processor -- start 2024-03-29T10:46:29.059 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.076+0000 7fe82d6a1640 1 -- start start 2024-03-29T10:46:29.060 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.076+0000 7fe82d6a1640 10 monclient: init 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-c con 0x7fe8281457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe82813f650 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-a con 0x7fe8280a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8280aaef0 con 0x7fe8280a9d70 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828145bd0 con 0x7fe8281457f0 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828145eb0 con 0x7fe82813f650 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.077+0000 7fe82d6a1640 10 monclient(hunting): authenticate will time out at 1520.246713s 2024-03-29T10:46:29.063 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.078+0000 7fe8267fc640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe82813f650 0x7fe8280b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:44528/0 (socket says 172.21.15.151:44528) 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.078+0000 7fe8267fc640 1 -- 172.21.15.151:0/2837997256 learned_addr learned my addr 172.21.15.151:0/2837997256 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115084129 0 0) 0x7fe828145eb0 con 0x7fe82813f650 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:46:29.064 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.079+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe810003520 con 0x7fe82813f650 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269515170 0 0) 0x7fe8280aaef0 con 0x7fe8280a9d70 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe828145eb0 con 0x7fe8280a9d70 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494212891 0 0) 0x7fe828145eb0 con 0x7fe8280a9d70 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 >> v1:172.21.15.151:6790/0 conn(0x7fe82813f650 legacy=0x7fe8280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.068 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 >> v1:172.21.15.169:6789/0 conn(0x7fe8281457f0 legacy=0x7fe82813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 monclient: found mon.noname-a 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.080+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828146340 con 0x7fe8280a9d70 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe82d6a1640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe82d6a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe8200019f0 con 0x7fe8280a9d70 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: dump: 2024-03-29T10:46:29.069 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe82d6a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe820001c10 con 0x7fe8280a9d70 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:29.070 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 1 -- 172.21.15.151:0/2837997256 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe820001fa0 con 0x7fe8280a9d70 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe825ffb640 10 monclient: dump: 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe82d6a1640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:29.071 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.081+0000 7fe82d6a1640 4 set_mon_vals no callback set 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.082+0000 7fe82d6a1640 10 monclient: shutdown 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.082+0000 7fe82d6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.082+0000 7fe82d6a1640 1 -- 172.21.15.151:0/2837997256 >> v1:172.21.15.151:6789/0 conn(0x7fe8280a9d70 legacy=0x7fe82814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.082+0000 7fe82d6a1640 1 -- 172.21.15.151:0/2837997256 shutdown_connections 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.082+0000 7fe82d6a1640 1 -- 172.21.15.151:0/2837997256 >> 172.21.15.151:0/2837997256 conn(0x7fe828066a00 msgr2=0x7fe828066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.084+0000 7fe82d6a1640 1 -- 172.21.15.151:0/2837997256 shutdown_connections 2024-03-29T10:46:29.072 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.085+0000 7fe82d6a1640 1 -- 172.21.15.151:0/2837997256 wait complete. 2024-03-29T10:46:29.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.086+0000 7fe82d6a1640 10 monclient: build_initial_monmap 2024-03-29T10:46:29.074 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.086+0000 7fe82d6a1640 10 monclient: monmap: 2024-03-29T10:46:29.074 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 librados: starting msgr at 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 librados: starting objecter 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 Processor -- start 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 -- start start 2024-03-29T10:46:29.075 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 librados: setting wanted keys 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 1 librados: calling monclient init 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.087+0000 7fe82d6a1640 10 monclient: init 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-c con 0x7fe8280a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe82813f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient(hunting): picked mon.noname-a con 0x7fe8281457f0 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828142030 con 0x7fe8281457f0 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82813b8a0 con 0x7fe82813f650 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8281422a0 con 0x7fe8280a9d70 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:29.076 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.088+0000 7fe82d6a1640 10 monclient(hunting): authenticate will time out at 1520.258213s 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.089+0000 7fe826ffd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fe8280a9d70 0x7fe82813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:48512/0 (socket says 172.21.15.151:48512) 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.089+0000 7fe826ffd640 1 -- 172.21.15.151:0/3215404842 learned_addr learned my addr 172.21.15.151:0/3215404842 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 client.?.objecter ms_handle_connect 0x7fe8280a9d70 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951530221 0 0) 0x7fe8281422a0 con 0x7fe8280a9d70 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814004eb0 con 0x7fe8280a9d70 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330377277 0 0) 0x7fe814004eb0 con 0x7fe8280a9d70 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe814005300 con 0x7fe8280a9d70 2024-03-29T10:46:29.077 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe818003fa0 con 0x7fe8280a9d70 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.078 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.090+0000 7fe80f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 client.?.objecter ms_handle_connect 0x7fe82813f650 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1101382309 0 0) 0x7fe814005300 con 0x7fe8280a9d70 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 >> v1:172.21.15.169:6789/0 conn(0x7fe82813f650 legacy=0x7fe8281e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 >> v1:172.21.15.151:6789/0 conn(0x7fe8281457f0 legacy=0x7fe8281e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 1 monclient: found mon.c 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828139f30 con 0x7fe8280a9d70 2024-03-29T10:46:29.081 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe80f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 10 monclient: _renew_subs 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe828071fc0 con 0x7fe8280a9d70 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 10 monclient: _renew_subs 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82813a450 con 0x7fe8280a9d70 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.093+0000 7fe82d6a1640 1 librados: init done 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe818000ed0 con 0x7fe8280a9d70 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:29.082 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fe818003fa0 con 0x7fe8280a9d70 2024-03-29T10:46:29.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:29.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:29.084 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80f7fe640 10 monclient: dump: 2024-03-29T10:46:29.084 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:29.084 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe828075f10 con 0x7fe8280a9d70 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.096+0000 7fe8257fa640 4 set_mon_vals no callback set 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.097+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fe818001180 con 0x7fe8280a9d70 2024-03-29T10:46:29.085 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7fe818054410 con 0x7fe8280a9d70 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe80f7fe640 10 client.4598.objecter ms_dispatch 0x7fe828000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe80f7fe640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe80f7fe640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe80f7fe640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.099+0000 7fe8267fc640 10 monclient: get_auth_request con 0x7fe81406e360 auth_method 0 2024-03-29T10:46:29.086 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.100+0000 7fe80f7fe640 10 client.4598.objecter ms_handle_connect 0x7fe81406e360 2024-03-29T10:46:29.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.107+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8180506b0 con 0x7fe8280a9d70 2024-03-29T10:46:29.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.107+0000 7fe80f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:29.090 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.107+0000 7fe80f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:29.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.451+0000 7fe80d7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.451+0000 7fe80d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.451+0000 7fe80d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:29.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.451+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8280b3680 con 0x7fe8280a9d70 2024-03-29T10:46:29.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.452+0000 7fe80f7fe640 1 -- 172.21.15.151:0/3215404842 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7fe8180506b0 con 0x7fe8280a9d70 2024-03-29T10:46:29.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.452+0000 7fe80f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.452+0000 7fe80f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:29.435 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:29.435 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:29.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 10 librados: watch_flush enter 2024-03-29T10:46:29.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 10 librados: watch_flush exit 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 10 client.4598.objecter successfully canceled tick 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 >> v1:172.21.15.151:6816/1549873000 conn(0x7fe81406e360 legacy=0x7fe814070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 10 monclient: shutdown 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 >> v1:172.21.15.151:6790/0 conn(0x7fe8280a9d70 legacy=0x7fe82813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 shutdown_connections 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 >> 172.21.15.151:0/3215404842 conn(0x7fe828066a00 msgr2=0x7fe8280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 shutdown_connections 2024-03-29T10:46:29.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 -- 172.21.15.151:0/3215404842 wait complete. 2024-03-29T10:46:29.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:29.457+0000 7fe80d7fa640 1 librados: shutdown 2024-03-29T10:46:29.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:36.265 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:36.405 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.423+0000 7f847f349640 10 monclient: get_monmap_and_config 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.423+0000 7f847f349640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.423+0000 7f847f349640 10 monclient: monmap: 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:36.424137+0000 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:36.424137+0000 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:36.406 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:36.407 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.426+0000 7f847f349640 1 Processor -- start 2024-03-29T10:46:36.409 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.426+0000 7f847f349640 1 -- start start 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.426+0000 7f847f349640 10 monclient: init 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient(hunting): picked mon.noname-a con 0x7f84780b2800 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient(hunting): picked mon.noname-c con 0x7f84780b3220 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient(hunting): picked mon.noname-b con 0x7f84781439c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84780b3840 con 0x7f84780b2800 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847806f230 con 0x7f84780b3220 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847806ef50 con 0x7f84781439c0 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.410 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.427+0000 7f847f349640 10 monclient(hunting): authenticate will time out at 1527.596960s 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.428+0000 7f847eb48640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f84781439c0 0x7f8478151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:55186/0 (socket says 172.21.15.151:55186) 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.428+0000 7f847eb48640 1 -- 172.21.15.151:0/3282503579 learned_addr learned my addr 172.21.15.151:0/3282503579 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374348376 0 0) 0x7f847806ef50 con 0x7f84781439c0 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.411 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:46:36.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8464003520 con 0x7f84781439c0 2024-03-29T10:46:36.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853506319 0 0) 0x7f8464003520 con 0x7f84781439c0 2024-03-29T10:46:36.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.429+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 >> v1:172.21.15.169:6789/0 conn(0x7f84780b3220 legacy=0x7f8478146710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:36.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 >> v1:172.21.15.151:6789/0 conn(0x7f84780b2800 legacy=0x7f84781432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:36.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: _finish_hunting 0 2024-03-29T10:46:36.415 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 1 monclient: found mon.noname-b 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478071d30 con 0x7f84781439c0 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: _finish_auth 0 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: _check_auth_tickets 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8474001690 con 0x7f84781439c0 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847d345640 10 monclient: dump: 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.416 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847f349640 5 monclient: authenticate success, global_id 4601 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.431+0000 7f847f349640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8474001860 con 0x7f84781439c0 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 1 -- 172.21.15.151:0/3282503579 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8474001cb0 con 0x7f84781439c0 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847d345640 10 monclient: dump: 2024-03-29T10:46:36.417 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847f349640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.432+0000 7f847f349640 4 set_mon_vals no callback set 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.433+0000 7f847f349640 10 monclient: shutdown 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.433+0000 7f847f349640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:36.418 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.433+0000 7f847f349640 1 -- 172.21.15.151:0/3282503579 >> v1:172.21.15.151:6790/0 conn(0x7f84781439c0 legacy=0x7f8478151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.434+0000 7f847f349640 1 -- 172.21.15.151:0/3282503579 shutdown_connections 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.434+0000 7f847f349640 1 -- 172.21.15.151:0/3282503579 >> 172.21.15.151:0/3282503579 conn(0x7f8478066a00 msgr2=0x7f8478066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.435+0000 7f847f349640 1 -- 172.21.15.151:0/3282503579 shutdown_connections 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.435+0000 7f847f349640 1 -- 172.21.15.151:0/3282503579 wait complete. 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 10 monclient: monmap: 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:36.420 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 librados: starting msgr at 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 librados: starting objecter 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 Processor -- start 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 -- start start 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 librados: setting wanted keys 2024-03-29T10:46:36.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 1 librados: calling monclient init 2024-03-29T10:46:36.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.436+0000 7f847f349640 10 monclient: init 2024-03-29T10:46:36.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.437+0000 7f847f349640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.422 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.437+0000 7f847f349640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:36.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.441+0000 7f847e347640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f84780b2800 0x7f84781de700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42473/0 (socket says 172.21.15.151:42473) 2024-03-29T10:46:36.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.441+0000 7f847e347640 1 -- 172.21.15.151:0/702140292 learned_addr learned my addr 172.21.15.151:0/702140292 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:36.423 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.441+0000 7f847f349640 10 monclient(hunting): picked mon.noname-a con 0x7f84780b2800 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:36.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.441+0000 7f847f349640 10 monclient(hunting): picked mon.noname-b con 0x7f84780b3220 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:36.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 10 monclient(hunting): picked mon.noname-c con 0x7f84781439c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:36.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84781dee10 con 0x7f84780b2800 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84780b5e90 con 0x7f84780b3220 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8478148d30 con 0x7f84781439c0 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f847f349640 10 monclient(hunting): authenticate will time out at 1527.613997s 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.442+0000 7f846effd640 10 client.?.objecter ms_handle_connect 0x7f84780b2800 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f846effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.444+0000 7f846effd640 10 client.?.objecter ms_handle_connect 0x7f84780b3220 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 client.?.objecter ms_handle_connect 0x7f84781439c0 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.433 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442217021 0 0) 0x7f84781dee10 con 0x7f84780b2800 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c004eb0 con 0x7f84780b2800 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047542108 0 0) 0x7f84780b5e90 con 0x7f84780b3220 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 10 monclient(hunting): my global_id is 4552 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.445+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84781dee10 con 0x7f84780b3220 2024-03-29T10:46:36.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134718677 0 0) 0x7f845c004eb0 con 0x7f84780b2800 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84780b5e90 con 0x7f84780b2800 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8470004400 con 0x7f84780b2800 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 10 monclient(hunting): dump: 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.435 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3730825455 0 0) 0x7f84781dee10 con 0x7f84780b3220 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f845c004eb0 con 0x7f84780b3220 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2703141118 0 0) 0x7f84780b5e90 con 0x7f84780b2800 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 >> v1:172.21.15.151:6790/0 conn(0x7f84781439c0 legacy=0x7f847813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 >> v1:172.21.15.169:6789/0 conn(0x7f84780b3220 legacy=0x7f84780b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.436 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.446+0000 7f846effd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 1 monclient: found mon.a 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847814a370 con 0x7f84780b2800 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 10 monclient: _finish_auth 0 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f846effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 10 monclient: _renew_subs 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847813c500 con 0x7f84780b2800 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 10 monclient: _renew_subs 2024-03-29T10:46:36.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.447+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f847813c220 con 0x7f84780b2800 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f847f349640 1 librados: init done 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8470004560 con 0x7f84780b2800 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f8470001500 con 0x7f84780b2800 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f846effd640 10 monclient: dump: 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:36.438 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:36.439 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:36.439 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:36.439 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:36.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.448+0000 7f847cb44640 4 set_mon_vals no callback set 2024-03-29T10:46:36.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.451+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f8470001660 con 0x7f84780b2800 2024-03-29T10:46:36.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.451+0000 7f847f349640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.451+0000 7f847f349640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.451+0000 7f847f349640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.451+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84781defa0 con 0x7f84780b2800 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.452+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f8470084db0 con 0x7f84780b2800 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.452+0000 7f847db46640 10 monclient: get_auth_request con 0x7f845c06de30 auth_method 0 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.453+0000 7f846effd640 10 client.4530.objecter ms_dispatch 0x7f8478000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.453+0000 7f846effd640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.453+0000 7f846effd640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.453+0000 7f846effd640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T10:46:36.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.453+0000 7f846effd640 10 client.4530.objecter ms_handle_connect 0x7f845c06de30 2024-03-29T10:46:36.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.461+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8470050690 con 0x7f84780b2800 2024-03-29T10:46:36.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.461+0000 7f846effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.461+0000 7f846effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:36.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.789+0000 7f847f349640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.789+0000 7f847f349640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.789+0000 7f847f349640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:36.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.789+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8478145920 con 0x7f84780b2800 2024-03-29T10:46:36.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.791+0000 7f846effd640 1 -- 172.21.15.151:0/702140292 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f84700525c0 con 0x7f84780b2800 2024-03-29T10:46:36.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.791+0000 7f846effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.791+0000 7f846effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:36.773 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:36.774 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.795+0000 7f847f349640 10 librados: watch_flush enter 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.795+0000 7f847f349640 10 librados: watch_flush exit 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.795+0000 7f847f349640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.795+0000 7f847f349640 10 client.4530.objecter successfully canceled tick 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 >> v1:172.21.15.151:6816/1549873000 conn(0x7f845c06de30 legacy=0x7f845c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 10 monclient: shutdown 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 >> v1:172.21.15.151:6789/0 conn(0x7f84780b2800 legacy=0x7f84781de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 shutdown_connections 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 >> 172.21.15.151:0/702140292 conn(0x7f8478066a00 msgr2=0x7f8478141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 shutdown_connections 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 -- 172.21.15.151:0/702140292 wait complete. 2024-03-29T10:46:36.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:36.796+0000 7f847f349640 1 librados: shutdown 2024-03-29T10:46:36.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:43.603 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:43.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.765+0000 7f73b280e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:43.747 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.765+0000 7f73b280e640 10 monclient: build_initial_monmap 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.765+0000 7f73b280e640 10 monclient: monmap: 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:43.765973+0000 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:43.765973+0000 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:43.748 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.766+0000 7f73b280e640 1 Processor -- start 2024-03-29T10:46:43.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.766+0000 7f73b280e640 1 -- start start 2024-03-29T10:46:43.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.766+0000 7f73b280e640 10 monclient: init 2024-03-29T10:46:43.762 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-c con 0x7f73ac1457f0 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-a con 0x7f73ac13f650 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-b con 0x7f73ac0a9d70 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73abfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f73ac1457f0 0x7f73ac13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:48200/0 (socket says 172.21.15.151:48200) 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73abfff640 1 -- 172.21.15.151:0/1736906907 learned_addr learned my addr 172.21.15.151:0/1736906907 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0aaef0 con 0x7f73ac13f650 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac145bd0 con 0x7f73ac1457f0 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac145eb0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.767+0000 7f73b280e640 10 monclient(hunting): authenticate will time out at 1534.937359s 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548031884 0 0) 0x7f73ac145bd0 con 0x7f73ac1457f0 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient(hunting): my global_id is 4555 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7390003520 con 0x7f73ac1457f0 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702502979 0 0) 0x7f7390003520 con 0x7f73ac1457f0 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 >> v1:172.21.15.151:6790/0 conn(0x7f73ac0a9d70 legacy=0x7f73ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 >> v1:172.21.15.151:6789/0 conn(0x7f73ac13f650 legacy=0x7f73ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 monclient: found mon.noname-c 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac146340 con 0x7f73ac1457f0 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f73a4001a10 con 0x7f73ac1457f0 2024-03-29T10:46:43.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73aaffd640 10 monclient: dump: 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.766 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.768+0000 7f73b280e640 5 monclient: authenticate success, global_id 4555 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73b280e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73a4001ca0 con 0x7f73ac1457f0 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 1 -- 172.21.15.151:0/1736906907 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f73a4002030 con 0x7f73ac1457f0 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:43.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73aaffd640 10 monclient: dump: 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.768 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73b280e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.769+0000 7f73b280e640 4 set_mon_vals no callback set 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.770+0000 7f73b280e640 10 monclient: shutdown 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.770+0000 7f73b280e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.770+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 >> v1:172.21.15.169:6789/0 conn(0x7f73ac1457f0 legacy=0x7f73ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.770+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 shutdown_connections 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.770+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 >> 172.21.15.151:0/1736906907 conn(0x7f73ac066a00 msgr2=0x7f73ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.771+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 shutdown_connections 2024-03-29T10:46:43.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.771+0000 7f73b280e640 1 -- 172.21.15.151:0/1736906907 wait complete. 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.772+0000 7f73b280e640 10 monclient: build_initial_monmap 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.772+0000 7f73b280e640 10 monclient: monmap: 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:43.770 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.772+0000 7f73b280e640 1 librados: starting msgr at 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.772+0000 7f73b280e640 1 librados: starting objecter 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.772+0000 7f73b280e640 1 Processor -- start 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.773+0000 7f73b280e640 1 -- start start 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.773+0000 7f73b280e640 1 librados: setting wanted keys 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.773+0000 7f73b280e640 1 librados: calling monclient init 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.773+0000 7f73b280e640 10 monclient: init 2024-03-29T10:46:43.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-a con 0x7f73ac0a9d70 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-b con 0x7f73ac13f650 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient(hunting): picked mon.noname-c con 0x7f73ac1457f0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac06e030 con 0x7f73ac0a9d70 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac06f790 con 0x7f73ac13f650 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac06fa70 con 0x7f73ac1457f0 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73b280e640 10 monclient(hunting): authenticate will time out at 1534.943856s 2024-03-29T10:46:43.772 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73abfff640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f73ac0a9d70 0x7f73ac140290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:60822/0 (socket says 172.21.15.151:60822) 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73abfff640 1 -- 172.21.15.151:0/614321452 learned_addr learned my addr 172.21.15.151:0/614321452 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 client.?.objecter ms_handle_connect 0x7f73ac0a9d70 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 client.?.objecter ms_handle_connect 0x7f73ac1457f0 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672504896 0 0) 0x7f73ac06e030 con 0x7f73ac0a9d70 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.773 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.774+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7394004eb0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 client.?.objecter ms_handle_connect 0x7f73ac13f650 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586489548 0 0) 0x7f73ac06fa70 con 0x7f73ac1457f0 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ac06e030 con 0x7f73ac1457f0 2024-03-29T10:46:43.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849000343 0 0) 0x7f7394004eb0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73ac06fa70 con 0x7f73ac0a9d70 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f73a4001990 con 0x7f73ac0a9d70 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.775 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.775+0000 7f73a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565921103 0 0) 0x7f73ac06f790 con 0x7f73ac13f650 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.776 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7394004eb0 con 0x7f73ac13f650 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455243082 0 0) 0x7f7394004eb0 con 0x7f73ac13f650 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73ac06f790 con 0x7f73ac13f650 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f7398001510 con 0x7f73ac13f650 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.777 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4255356271 0 0) 0x7f73ac06e030 con 0x7f73ac1457f0 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7394004eb0 con 0x7f73ac1457f0 2024-03-29T10:46:43.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f739c001830 con 0x7f73ac1457f0 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.779 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3533767091 0 0) 0x7f73ac06fa70 con 0x7f73ac0a9d70 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 >> v1:172.21.15.151:6790/0 conn(0x7f73ac1457f0 legacy=0x7f73ac1e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 >> v1:172.21.15.169:6789/0 conn(0x7f73ac13f650 legacy=0x7f73ac1e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:43.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 monclient: found mon.a 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac06fd50 con 0x7f73ac0a9d70 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 10 monclient: _renew_subs 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:43.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac1409a0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 10 monclient: _renew_subs 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac142d10 con 0x7f73ac0a9d70 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.776+0000 7f73b280e640 1 librados: init done 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73a4004e10 con 0x7f73ac0a9d70 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f73a4005120 con 0x7f73ac0a9d70 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a8ff9640 10 monclient: dump: 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:43.782 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.777+0000 7f73a9ffb640 4 set_mon_vals no callback set 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.779+0000 7f73b280e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.779+0000 7f73b280e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.779+0000 7f73b280e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:43.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.779+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388005080 con 0x7f73ac0a9d70 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.780+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f73a401b0f0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f73a4084e00 con 0x7f73ac0a9d70 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 10 client.4533.objecter ms_dispatch 0x7f73ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73ab7fe640 10 monclient: get_auth_request con 0x7f739406e010 auth_method 0 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.782+0000 7f73a8ff9640 10 client.4533.objecter ms_handle_connect 0x7f739406e010 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.789+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73a40506e0 con 0x7f73ac0a9d70 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.789+0000 7f73a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:43.789+0000 7f73a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:44.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.162+0000 7f73b280e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.163+0000 7f73b280e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.163+0000 7f73b280e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:44.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.163+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7388005330 con 0x7f73ac0a9d70 2024-03-29T10:46:44.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.163+0000 7f73a8ff9640 1 -- 172.21.15.151:0/614321452 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f73a4052610 con 0x7f73ac0a9d70 2024-03-29T10:46:44.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.164+0000 7f73a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.146 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.164+0000 7f73a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:44.146 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:44.146 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:44.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 10 librados: watch_flush enter 2024-03-29T10:46:44.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 10 librados: watch_flush exit 2024-03-29T10:46:44.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T10:46:44.149 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 10 client.4533.objecter successfully canceled tick 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 >> v1:172.21.15.151:6816/1549873000 conn(0x7f739406e010 legacy=0x7f73940704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 10 monclient: shutdown 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 >> v1:172.21.15.151:6789/0 conn(0x7f73ac0a9d70 legacy=0x7f73ac140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 shutdown_connections 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.167+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 >> 172.21.15.151:0/614321452 conn(0x7f73ac066a00 msgr2=0x7f73ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:44.150 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.168+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 shutdown_connections 2024-03-29T10:46:44.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.168+0000 7f73b280e640 1 -- 172.21.15.151:0/614321452 wait complete. 2024-03-29T10:46:44.151 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:44.168+0000 7f73b280e640 1 librados: shutdown 2024-03-29T10:46:44.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:50.971 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.126+0000 7f015fa0f640 10 monclient: get_monmap_and_config 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.126+0000 7f015fa0f640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.126+0000 7f015fa0f640 10 monclient: monmap: 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:51.127873+0000 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:51.127873+0000 2024-03-29T10:46:51.109 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.110 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.127+0000 7f015fa0f640 1 Processor -- start 2024-03-29T10:46:51.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.128+0000 7f015fa0f640 1 -- start start 2024-03-29T10:46:51.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.128+0000 7f015fa0f640 10 monclient: init 2024-03-29T10:46:51.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f015806ade0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f015806ed60 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f0158072340 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015806ff50 con 0x7f015806ed60 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0158070180 con 0x7f0158072340 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015806f310 con 0x7f015806ade0 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015fa0f640 10 monclient(hunting): authenticate will time out at 1542.298804s 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015dfac640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f0158072340 0x7f0158151d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:49468/0 (socket says 172.21.15.151:49468) 2024-03-29T10:46:51.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.129+0000 7f015dfac640 1 -- 172.21.15.151:0/301725260 learned_addr learned my addr 172.21.15.151:0/301725260 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:51.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335079288 0 0) 0x7f0158070180 con 0x7f0158072340 2024-03-29T10:46:51.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003520 con 0x7f0158072340 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503476106 0 0) 0x7f015806f310 con 0x7f015806ade0 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0158070180 con 0x7f015806ade0 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512114115 0 0) 0x7f0144003520 con 0x7f0158072340 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 >> v1:172.21.15.151:6790/0 conn(0x7f015806ade0 legacy=0x7f01581499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.131+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 >> v1:172.21.15.151:6789/0 conn(0x7f015806ed60 legacy=0x7f0158071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:51.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 1 monclient: found mon.noname-c 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580aeed0 con 0x7f0158072340 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0148001690 con 0x7f0158072340 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f015fa0f640 5 monclient: authenticate success, global_id 4561 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f015fa0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: dump: 2024-03-29T10:46:51.117 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01480018b0 con 0x7f0158072340 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 1 -- 172.21.15.151:0/301725260 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f0148001c40 con 0x7f0158072340 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f015fa0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f015fa0f640 4 set_mon_vals no callback set 2024-03-29T10:46:51.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.132+0000 7f014ffff640 10 monclient: dump: 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.133+0000 7f015fa0f640 10 monclient: shutdown 2024-03-29T10:46:51.119 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.133+0000 7f015fa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:51.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.133+0000 7f015fa0f640 1 -- 172.21.15.151:0/301725260 >> v1:172.21.15.169:6789/0 conn(0x7f0158072340 legacy=0x7f0158151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.133+0000 7f015fa0f640 1 -- 172.21.15.151:0/301725260 shutdown_connections 2024-03-29T10:46:51.120 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.133+0000 7f015fa0f640 1 -- 172.21.15.151:0/301725260 >> 172.21.15.151:0/301725260 conn(0x7f01580b4930 msgr2=0x7f01580b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:51.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.134+0000 7f015fa0f640 1 -- 172.21.15.151:0/301725260 shutdown_connections 2024-03-29T10:46:51.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.134+0000 7f015fa0f640 1 -- 172.21.15.151:0/301725260 wait complete. 2024-03-29T10:46:51.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 10 monclient: monmap: 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 librados: starting msgr at 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 librados: starting objecter 2024-03-29T10:46:51.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 Processor -- start 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 -- start start 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 librados: setting wanted keys 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 1 librados: calling monclient init 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.135+0000 7f015fa0f640 10 monclient: init 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f015806ade0 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f015806ed60 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f0158072340 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580ab650 con 0x7f0158072340 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580ab930 con 0x7f015806ed60 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01581472c0 con 0x7f015806ade0 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.136+0000 7f015fa0f640 10 monclient(hunting): authenticate will time out at 1542.306314s 2024-03-29T10:46:51.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f015d7ab640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f015806ade0 0x7f01580a9750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:48116/0 (socket says 172.21.15.151:48116) 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f015d7ab640 1 -- 172.21.15.151:0/3164984988 learned_addr learned my addr 172.21.15.151:0/3164984988 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 client.?.objecter ms_handle_connect 0x7f015806ade0 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189925612 0 0) 0x7f01581472c0 con 0x7f015806ade0 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.137+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c004eb0 con 0x7f015806ade0 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 client.?.objecter ms_handle_connect 0x7f015806ed60 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306521719 0 0) 0x7f013c004eb0 con 0x7f015806ade0 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f013c005300 con 0x7f015806ade0 2024-03-29T10:46:51.127 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f01540019f0 con 0x7f015806ade0 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): dump: 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:51.128 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844580658 0 0) 0x7f01580ab930 con 0x7f015806ed60 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c005880 con 0x7f015806ed60 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 705402832 0 0) 0x7f013c005300 con 0x7f015806ade0 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 >> v1:172.21.15.169:6789/0 conn(0x7f015806ed60 legacy=0x7f01581e4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 >> v1:172.21.15.151:6789/0 conn(0x7f0158072340 legacy=0x7f0158144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 monclient: found mon.c 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580aa760 con 0x7f015806ade0 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f014dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f015fa0f640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f015fa0f640 10 monclient: _renew_subs 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f015fa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.138+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01581e4730 con 0x7f015806ade0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f015fa0f640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f015fa0f640 10 monclient: _renew_subs 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f015fa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01581454f0 con 0x7f015806ade0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f015fa0f640 1 librados: init done 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0154001c50 con 0x7f015806ade0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f014dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.139+0000 7f014f7fe640 4 set_mon_vals no callback set 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f01540019f0 con 0x7f015806ade0 2024-03-29T10:46:51.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f014dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f014dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f014dffb640 10 monclient: dump: 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f015fa0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f015fa0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.131 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f015fa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:51.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.140+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0138005080 con 0x7f015806ade0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f01540038c0 con 0x7f015806ade0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1338434279 0 0) 0x7f0154084590 con 0x7f015806ade0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 10 client.4616.objecter ms_dispatch 0x7f0158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.143+0000 7f014dffb640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.144+0000 7f015cfaa640 10 monclient: get_auth_request con 0x7f013c06e430 auth_method 0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.144+0000 7f014dffb640 10 client.4616.objecter ms_handle_connect 0x7f013c06e430 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.149+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f015404fed0 con 0x7f015806ade0 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.149+0000 7f014dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.149+0000 7f014dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:51.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.451+0000 7f015fa0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.451+0000 7f015fa0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.451+0000 7f015fa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:46:51.434 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.451+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0138005330 con 0x7f015806ade0 2024-03-29T10:46:51.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.452+0000 7f014dffb640 1 -- 172.21.15.151:0/3164984988 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 20224549) 0x7f015404fed0 con 0x7f015806ade0 2024-03-29T10:46:51.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.452+0000 7f014dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.435 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.452+0000 7f014dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:51.435 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:51.435 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":10,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:36:54.842433+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:36:54.842433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:51.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 10 librados: watch_flush enter 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 10 librados: watch_flush exit 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 10 client.4616.objecter successfully canceled tick 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 >> v1:172.21.15.151:6816/1549873000 conn(0x7f013c06e430 legacy=0x7f013c0708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 10 monclient: shutdown 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 >> v1:172.21.15.151:6790/0 conn(0x7f015806ade0 legacy=0x7f01580a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 shutdown_connections 2024-03-29T10:46:51.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.456+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 >> 172.21.15.151:0/3164984988 conn(0x7f01580b4930 msgr2=0x7f01580afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:51.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.457+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 shutdown_connections 2024-03-29T10:46:51.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.457+0000 7f015fa0f640 1 -- 172.21.15.151:0/3164984988 wait complete. 2024-03-29T10:46:51.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:51.457+0000 7f015fa0f640 1 librados: shutdown 2024-03-29T10:46:51.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:58.262 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.430+0000 7f2dbfb8c640 10 monclient: get_monmap_and_config 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.430+0000 7f2dbfb8c640 10 monclient: build_initial_monmap 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.430+0000 7f2dbfb8c640 10 monclient: monmap: 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:46:58.431363+0000 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:46:58.431363+0000 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.434+0000 7f2dbfb8c640 1 Processor -- start 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.434+0000 7f2dbfb8c640 1 -- start start 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.434+0000 7f2dbfb8c640 10 monclient: init 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.435+0000 7f2dbfb8c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.435+0000 7f2dbfb8c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-c con 0x7f2db80b2520 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-a con 0x7f2db80b2900 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-b con 0x7f2db80a9c70 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:58.419 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db8142170 con 0x7f2db80b2900 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80b2260 con 0x7f2db80b2520 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80aaf60 con 0x7f2db80a9c70 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.436+0000 7f2dbfb8c640 10 monclient(hunting): authenticate will time out at 1549.605864s 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.437+0000 7f2dbe129640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f2db80a9c70 0x7f2db814d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:38054/0 (socket says 172.21.15.151:38054) 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.437+0000 7f2dbe129640 1 -- 172.21.15.151:0/2618743410 learned_addr learned my addr 172.21.15.151:0/2618743410 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190220978 0 0) 0x7f2db80aaf60 con 0x7f2db80a9c70 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 10 monclient(hunting): _negotiate 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da4003520 con 0x7f2db80a9c70 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725447963 0 0) 0x7f2da4003520 con 0x7f2db80a9c70 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.438+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 >> v1:172.21.15.169:6789/0 conn(0x7f2db80b2520 legacy=0x7f2db80b1b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 >> v1:172.21.15.151:6789/0 conn(0x7f2db80b2900 legacy=0x7f2db8142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: _finish_hunting 0 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 1 monclient: found mon.noname-b 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db8142d10 con 0x7f2db80a9c70 2024-03-29T10:46:58.424 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: _finish_auth 0 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: _check_auth_tickets 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2db4001690 con 0x7f2db80a9c70 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbc926640 10 monclient: dump: 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:58.425 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbfb8c640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.440+0000 7f2dbfb8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db4001860 con 0x7f2db80a9c70 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 1 -- 172.21.15.151:0/2618743410 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2db4001c70 con 0x7f2db80a9c70 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbc926640 10 monclient: dump: 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:58.426 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbfb8c640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:58.427 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.441+0000 7f2dbfb8c640 4 set_mon_vals no callback set 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.442+0000 7f2dbfb8c640 10 monclient: shutdown 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.442+0000 7f2dbfb8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.442+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/2618743410 >> v1:172.21.15.151:6790/0 conn(0x7f2db80a9c70 legacy=0x7f2db814d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.442+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/2618743410 shutdown_connections 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.442+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/2618743410 >> 172.21.15.151:0/2618743410 conn(0x7f2db806d180 msgr2=0x7f2db8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.443+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/2618743410 shutdown_connections 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.443+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/2618743410 wait complete. 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.444+0000 7f2dbfb8c640 10 monclient: build_initial_monmap 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.444+0000 7f2dbfb8c640 10 monclient: monmap: 2024-03-29T10:46:58.430 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.444+0000 7f2dbfb8c640 1 librados: starting msgr at 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.444+0000 7f2dbfb8c640 1 librados: starting objecter 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.445+0000 7f2dbfb8c640 1 Processor -- start 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.445+0000 7f2dbfb8c640 1 -- start start 2024-03-29T10:46:58.431 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.445+0000 7f2dbfb8c640 1 librados: setting wanted keys 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.445+0000 7f2dbfb8c640 1 librados: calling monclient init 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.445+0000 7f2dbfb8c640 10 monclient: init 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.446+0000 7f2dbfb8c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.446+0000 7f2dbfb8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.446+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-a con 0x7f2db80a9c70 addr v1:172.21.15.151:6789/0 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-b con 0x7f2db80b2520 addr v1:172.21.15.169:6789/0 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 10 monclient(hunting): picked mon.noname-c con 0x7f2db8145700 addr v1:172.21.15.151:6790/0 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80abbe0 con 0x7f2db80a9c70 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db813e380 con 0x7f2db80b2520 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db8137340 con 0x7f2db8145700 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:58.432 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.447+0000 7f2dbfb8c640 10 monclient(hunting): authenticate will time out at 1549.617096s 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.449+0000 7f2dbe129640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f2db8145700 0x7f2db81d6e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:38938/0 (socket says 172.21.15.151:38938) 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.449+0000 7f2dbe129640 1 -- 172.21.15.151:0/1176733765 learned_addr learned my addr 172.21.15.151:0/1176733765 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.449+0000 7f2dae7fc640 10 client.?.objecter ms_handle_connect 0x7f2db8145700 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.449+0000 7f2dae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 client.?.objecter ms_handle_connect 0x7f2db80a9c70 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716864152 0 0) 0x7f2db80abbe0 con 0x7f2db80a9c70 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:58.437 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.450+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9c004eb0 con 0x7f2db80a9c70 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208845133 0 0) 0x7f2db8137340 con 0x7f2db8145700 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db80abbe0 con 0x7f2db8145700 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666862792 0 0) 0x7f2d9c004eb0 con 0x7f2db80a9c70 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db8137340 con 0x7f2db80a9c70 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2db0001b50 con 0x7f2db80a9c70 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:58.438 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): dump: 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.451+0000 7f2dae7fc640 10 client.?.objecter ms_handle_connect 0x7f2db80b2520 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:58.439 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152909035 0 0) 0x7f2db80abbe0 con 0x7f2db8145700 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d9c004eb0 con 0x7f2db8145700 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2485328304 0 0) 0x7f2db8137340 con 0x7f2db80a9c70 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 >> v1:172.21.15.151:6790/0 conn(0x7f2db8145700 legacy=0x7f2db81d6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 >> v1:172.21.15.169:6789/0 conn(0x7f2db80b2520 legacy=0x7f2db8136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 monclient: found mon.a 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db81374d0 con 0x7f2db80a9c70 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 5 monclient: authenticate success, global_id 4539 2024-03-29T10:46:58.440 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 10 monclient: _renew_subs 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db8137740 con 0x7f2db80a9c70 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 10 monclient: _renew_subs 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db81d7770 con 0x7f2db80a9c70 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.452+0000 7f2dbfb8c640 1 librados: init done 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.453+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db0002060 con 0x7f2db80a9c70 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.453+0000 7f2dae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.453+0000 7f2daffff640 4 set_mon_vals no callback set 2024-03-29T10:46:58.441 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.454+0000 7f2dbfb8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.455+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f2db0003b40 con 0x7f2db80a9c70 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.455+0000 7f2dae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.455+0000 7f2dae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.455+0000 7f2dae7fc640 10 monclient: dump: 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.443 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.456+0000 7f2dbfb8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.456+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f2db0019b10 con 0x7f2db80a9c70 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.456+0000 7f2dbfb8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.456+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da0005080 con 0x7f2db80a9c70 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7f2db00839c0 con 0x7f2db80a9c70 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dbd127640 10 monclient: get_auth_request con 0x7f2d9c06dd20 auth_method 0 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 10 client.4539.objecter ms_dispatch 0x7f2db8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:46:58.444 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 3 client.4539.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:46:58.445 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 3 client.4539.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:46:58.445 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T10:46:58.445 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.457+0000 7f2dae7fc640 10 client.4539.objecter ms_handle_connect 0x7f2d9c06dd20 2024-03-29T10:46:58.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.466+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2db004f300 con 0x7f2db80a9c70 2024-03-29T10:46:58.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.466+0000 7f2dae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.449 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.466+0000 7f2dae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dbfb8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dbfb8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dbfb8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2da0005330 con 0x7f2db80a9c70 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dae7fc640 1 -- 172.21.15.151:0/1176733765 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7f2db0083070 con 0x7f2db80a9c70 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.774 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.791+0000 7f2dae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:58.775 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:46:58.775 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:58.777 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.795+0000 7f2dbfb8c640 10 librados: watch_flush enter 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 10 librados: watch_flush exit 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 10 client.4539.objecter successfully canceled tick 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 >> v1:172.21.15.151:6816/1549873000 conn(0x7f2d9c06dd20 legacy=0x7f2d9c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 10 monclient: shutdown 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 >> v1:172.21.15.151:6789/0 conn(0x7f2db80a9c70 legacy=0x7f2db813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 shutdown_connections 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 >> 172.21.15.151:0/1176733765 conn(0x7f2db806d180 msgr2=0x7f2db81401b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 shutdown_connections 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 -- 172.21.15.151:0/1176733765 wait complete. 2024-03-29T10:46:58.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:46:58.796+0000 7f2dbfb8c640 1 librados: shutdown 2024-03-29T10:46:58.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:05.602 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.773+0000 7fbe5f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.773+0000 7fbe5f59e640 10 monclient: build_initial_monmap 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.773+0000 7fbe5f59e640 10 monclient: monmap: 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:05.774882+0000 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:05.774882+0000 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.775+0000 7fbe5f59e640 1 Processor -- start 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.775+0000 7fbe5f59e640 1 -- start start 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.775+0000 7fbe5f59e640 10 monclient: init 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe60143a90 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe60146fd0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5dd9b640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fbe60146fd0 0x7fbe600b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60550/0 (socket says 172.21.15.151:60550) 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5dd9b640 1 -- 172.21.15.151:0/81866377 learned_addr learned my addr 172.21.15.151:0/81866377 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe600b2e50 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6006ef90 con 0x7fbe60143a90 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6006f1c0 con 0x7fbe600b2e50 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe60071eb0 con 0x7fbe60146fd0 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.776+0000 7fbe5f59e640 10 monclient(hunting): authenticate will time out at 1556.946310s 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123082155 0 0) 0x7fbe60071eb0 con 0x7fbe60146fd0 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe48003520 con 0x7fbe60146fd0 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.777+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373007006 0 0) 0x7fbe48003520 con 0x7fbe60146fd0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 >> v1:172.21.15.169:6789/0 conn(0x7fbe600b2e50 legacy=0x7fbe60151c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 >> v1:172.21.15.151:6789/0 conn(0x7fbe60143a90 legacy=0x7fbe60146320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 monclient: found mon.noname-b 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe600b3300 con 0x7fbe60146fd0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: _finish_auth 0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbe54001690 con 0x7fbe60146fd0 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: dump: 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5f59e640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe54001860 con 0x7fbe60146fd0 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:05.765 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 1 -- 172.21.15.151:0/81866377 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbe54001cb0 con 0x7fbe60146fd0 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5d59a640 10 monclient: dump: 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:05.766 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.778+0000 7fbe5f59e640 4 set_mon_vals no callback set 2024-03-29T10:47:05.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.779+0000 7fbe5f59e640 10 monclient: shutdown 2024-03-29T10:47:05.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.779+0000 7fbe5f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:05.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.779+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 >> v1:172.21.15.151:6790/0 conn(0x7fbe60146fd0 legacy=0x7fbe600b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.781+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 shutdown_connections 2024-03-29T10:47:05.767 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.781+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 >> 172.21.15.151:0/81866377 conn(0x7fbe6006d0e0 msgr2=0x7fbe60066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.782+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 shutdown_connections 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.782+0000 7fbe5f59e640 1 -- 172.21.15.151:0/81866377 wait complete. 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.783+0000 7fbe5f59e640 10 monclient: build_initial_monmap 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.783+0000 7fbe5f59e640 10 monclient: monmap: 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.783+0000 7fbe5f59e640 1 librados: starting msgr at 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.783+0000 7fbe5f59e640 1 librados: starting objecter 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.784+0000 7fbe5f59e640 1 Processor -- start 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.784+0000 7fbe5f59e640 1 -- start start 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.784+0000 7fbe5f59e640 1 librados: setting wanted keys 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.784+0000 7fbe5f59e640 1 librados: calling monclient init 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.784+0000 7fbe5f59e640 10 monclient: init 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.785+0000 7fbe5f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.785+0000 7fbe5f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.785+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe600b2e50 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe60143a90 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe60146fd0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe601debc0 con 0x7fbe60146fd0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe600b8760 con 0x7fbe600b2e50 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6013d860 con 0x7fbe60143a90 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.786+0000 7fbe5f59e640 10 monclient(hunting): authenticate will time out at 1556.956161s 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe5ed9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbe60146fd0 0x7fbe6013d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:37239/0 (socket says 172.21.15.151:37239) 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe5dd9b640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fbe60143a90 0x7fbe6013ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:33580/0 (socket says 172.21.15.151:33580) 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe5dd9b640 1 -- 172.21.15.151:0/3514127479 learned_addr learned my addr 172.21.15.151:0/3514127479 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 client.?.objecter ms_handle_connect 0x7fbe60143a90 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.778 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 client.?.objecter ms_handle_connect 0x7fbe60146fd0 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192952111 0 0) 0x7fbe601debc0 con 0x7fbe60146fd0 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.790+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe40004eb0 con 0x7fbe60146fd0 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 client.?.objecter ms_handle_connect 0x7fbe600b2e50 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509774898 0 0) 0x7fbe6013d860 con 0x7fbe60143a90 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.779 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe601debc0 con 0x7fbe60143a90 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 789879072 0 0) 0x7fbe40004eb0 con 0x7fbe60146fd0 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe6013d860 con 0x7fbe60146fd0 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbe58001770 con 0x7fbe60146fd0 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): dump: 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.780 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528075232 0 0) 0x7fbe600b8760 con 0x7fbe600b2e50 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.781 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe40004eb0 con 0x7fbe600b2e50 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.791+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087528908 0 0) 0x7fbe601debc0 con 0x7fbe60143a90 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe600b8760 con 0x7fbe60143a90 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbe54001b70 con 0x7fbe60143a90 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient(hunting): dump: 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.782 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448050163 0 0) 0x7fbe6013d860 con 0x7fbe60146fd0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 >> v1:172.21.15.151:6790/0 conn(0x7fbe60143a90 legacy=0x7fbe6013ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 >> v1:172.21.15.169:6789/0 conn(0x7fbe600b2e50 legacy=0x7fbe601df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 monclient: found mon.a 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6014ad30 con 0x7fbe60146fd0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient: _finish_auth 0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 5 monclient: authenticate success, global_id 4542 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 10 monclient: _renew_subs 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe601de7c0 con 0x7fbe60146fd0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 10 monclient: _renew_subs 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe601495f0 con 0x7fbe60146fd0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.792+0000 7fbe5f59e640 1 librados: init done 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe58001c80 con 0x7fbe60146fd0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fbe58004a00 con 0x7fbe60146fd0 2024-03-29T10:47:05.784 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 10 monclient: dump: 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.794+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fbe5801a9d0 con 0x7fbe60146fd0 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.795+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7fbe58084990 con 0x7fbe60146fd0 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.795+0000 7fbe5e59c640 10 monclient: get_auth_request con 0x7fbe4006df10 auth_method 0 2024-03-29T10:47:05.785 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe5cd99640 4 set_mon_vals no callback set 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe5f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe4effd640 10 client.4542.objecter ms_dispatch 0x7fbe60000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe4effd640 3 client.4542.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe4effd640 3 client.4542.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.796+0000 7fbe4effd640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.797+0000 7fbe5f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.786 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.797+0000 7fbe5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:05.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.797+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe601ded70 con 0x7fbe60146fd0 2024-03-29T10:47:05.787 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.797+0000 7fbe4effd640 10 client.4542.objecter ms_handle_connect 0x7fbe4006df10 2024-03-29T10:47:05.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.805+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe601ded70 con 0x7fbe60146fd0 2024-03-29T10:47:05.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.805+0000 7fbe4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.788 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:05.805+0000 7fbe4effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:06.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.134+0000 7fbe5f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.134+0000 7fbe5f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.134+0000 7fbe5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:06.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.134+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe601ded70 con 0x7fbe60146fd0 2024-03-29T10:47:06.117 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.135+0000 7fbe4effd640 1 -- 172.21.15.151:0/3514127479 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7fbe601ded70 con 0x7fbe60146fd0 2024-03-29T10:47:06.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.135+0000 7fbe4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.135+0000 7fbe4effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:06.118 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:06.118 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 10 librados: watch_flush enter 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 10 librados: watch_flush exit 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 10 client.4542.objecter successfully canceled tick 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 >> v1:172.21.15.151:6816/1549873000 conn(0x7fbe4006df10 legacy=0x7fbe400703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 10 monclient: shutdown 2024-03-29T10:47:06.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.139+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 >> v1:172.21.15.151:6789/0 conn(0x7fbe60146fd0 legacy=0x7fbe6013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.140+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 shutdown_connections 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.140+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 >> 172.21.15.151:0/3514127479 conn(0x7fbe6006d0e0 msgr2=0x7fbe600b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.140+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 shutdown_connections 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.140+0000 7fbe5f59e640 1 -- 172.21.15.151:0/3514127479 wait complete. 2024-03-29T10:47:06.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:06.140+0000 7fbe5f59e640 1 librados: shutdown 2024-03-29T10:47:06.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:12.947 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:13.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.117+0000 7f18ca934640 10 monclient: get_monmap_and_config 2024-03-29T10:47:13.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.117+0000 7f18ca934640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.100 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.118+0000 7f18ca934640 10 monclient: monmap: 2024-03-29T10:47:13.100 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:13.118952+0000 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:13.118952+0000 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.120+0000 7f18ca934640 1 Processor -- start 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.120+0000 7f18ca934640 1 -- start start 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.120+0000 7f18ca934640 10 monclient: init 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.121+0000 7f18ca934640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.121+0000 7f18ca934640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.121+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-a con 0x7f18c40b2800 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.122+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-b con 0x7f18c40a9c00 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-c con 0x7f18c40b38c0 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c4141eb0 con 0x7f18c40b2800 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40b22d0 con 0x7f18c40b38c0 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c4142d30 con 0x7f18c40a9c00 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.123+0000 7f18ca934640 10 monclient(hunting): authenticate will time out at 1564.292580s 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c9131640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f18c40a9c00 0x7f18c413bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:59076/0 (socket says 172.21.15.151:59076) 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c9131640 1 -- 172.21.15.151:0/3434115344 learned_addr learned my addr 172.21.15.151:0/3434115344 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293159264 0 0) 0x7f18c4142d30 con 0x7f18c40a9c00 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.124+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b0003520 con 0x7f18c40a9c00 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124473819 0 0) 0x7f18c4141eb0 con 0x7f18c40b2800 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4142d30 con 0x7f18c40b2800 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311178288 0 0) 0x7f18b0003520 con 0x7f18c40a9c00 2024-03-29T10:47:13.109 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.125+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 >> v1:172.21.15.169:6789/0 conn(0x7f18c40b38c0 legacy=0x7f18c414d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:13.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 >> v1:172.21.15.151:6789/0 conn(0x7f18c40b2800 legacy=0x7f18c4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 1 monclient: found mon.noname-b 2024-03-29T10:47:13.112 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40aaf90 con 0x7f18c40a9c00 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f18c0001690 con 0x7f18c40a9c00 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18c8930640 10 monclient: dump: 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18ca934640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.126+0000 7f18ca934640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18c0001870 con 0x7f18c40a9c00 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 1 -- 172.21.15.151:0/3434115344 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f18c0001c40 con 0x7f18c40a9c00 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18c8930640 10 monclient: dump: 2024-03-29T10:47:13.114 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18ca934640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.129+0000 7f18ca934640 4 set_mon_vals no callback set 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.130+0000 7f18ca934640 10 monclient: shutdown 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.130+0000 7f18ca934640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:13.115 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.130+0000 7f18ca934640 1 -- 172.21.15.151:0/3434115344 >> v1:172.21.15.151:6790/0 conn(0x7f18c40a9c00 legacy=0x7f18c413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.130+0000 7f18ca934640 1 -- 172.21.15.151:0/3434115344 shutdown_connections 2024-03-29T10:47:13.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.130+0000 7f18ca934640 1 -- 172.21.15.151:0/3434115344 >> 172.21.15.151:0/3434115344 conn(0x7f18c4066a00 msgr2=0x7f18c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:13.116 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.132+0000 7f18ca934640 1 -- 172.21.15.151:0/3434115344 shutdown_connections 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.134+0000 7f18ca934640 1 -- 172.21.15.151:0/3434115344 wait complete. 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.135+0000 7f18ca934640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.135+0000 7f18ca934640 10 monclient: monmap: 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:13.118 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:13.119 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.136+0000 7f18ca934640 1 librados: starting msgr at 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.136+0000 7f18ca934640 1 librados: starting objecter 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.136+0000 7f18ca934640 1 Processor -- start 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.136+0000 7f18ca934640 1 -- start start 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.136+0000 7f18ca934640 1 librados: setting wanted keys 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.137+0000 7f18ca934640 1 librados: calling monclient init 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.137+0000 7f18ca934640 10 monclient: init 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-b con 0x7f18c40a9c00 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-a con 0x7f18c40b2800 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:13.121 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient(hunting): picked mon.noname-c con 0x7f18c40b38c0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c41da270 con 0x7f18c40b2800 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40afc20 con 0x7f18c40a9c00 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c41da510 con 0x7f18c40b38c0 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.138+0000 7f18ca934640 10 monclient(hunting): authenticate will time out at 1564.308352s 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ca133640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f18c40b38c0 0x7f18c40b0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:60068/0 (socket says 172.21.15.151:60068) 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ca133640 1 -- 172.21.15.151:0/3010716554 learned_addr learned my addr 172.21.15.151:0/3010716554 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:13.122 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 client.?.objecter ms_handle_connect 0x7f18c40b38c0 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468802029 0 0) 0x7f18c41da510 con 0x7f18c40b38c0 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:47:13.123 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.139+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8004eb0 con 0x7f18c40b38c0 2024-03-29T10:47:13.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077539993 0 0) 0x7f18a8004eb0 con 0x7f18c40b38c0 2024-03-29T10:47:13.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18a8005300 con 0x7f18c40b38c0 2024-03-29T10:47:13.124 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f18bc0042a0 con 0x7f18c40b38c0 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 10 monclient(hunting): dump: 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.125 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:13.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1882516320 0 0) 0x7f18a8005300 con 0x7f18c40b38c0 2024-03-29T10:47:13.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.141+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 >> v1:172.21.15.169:6789/0 conn(0x7f18c40a9c00 legacy=0x7f18c40ad5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:13.126 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.142+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 >> v1:172.21.15.151:6789/0 conn(0x7f18c40b2800 legacy=0x7f18c41daa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:13.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.129 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 1 monclient: found mon.c 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c41db140 con 0x7f18c40b38c0 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18bc004400 con 0x7f18c40b38c0 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f18bc0014b0 con 0x7f18c40b38c0 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:13.130 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ba7fc640 10 monclient: dump: 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:13.131 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ca934640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ca934640 10 monclient: _renew_subs 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.147+0000 7f18ca934640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c40adfa0 con 0x7f18c40b38c0 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 10 monclient: _renew_subs 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c40b1230 con 0x7f18c40b38c0 2024-03-29T10:47:13.132 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.148+0000 7f18ca934640 1 librados: init done 2024-03-29T10:47:13.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.149+0000 7f18bbfff640 4 set_mon_vals no callback set 2024-03-29T10:47:13.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.150+0000 7f18ca934640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.133 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f18bc001690 con 0x7f18c40b38c0 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ca934640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ca934640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac005080 con 0x7f18c40b38c0 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7f18bc084ae0 con 0x7f18c40b38c0 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 10 client.4634.objecter ms_dispatch 0x7f18c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:13.134 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.151+0000 7f18ba7fc640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T10:47:13.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.152+0000 7f18c9932640 10 monclient: get_auth_request con 0x7f18a806df90 auth_method 0 2024-03-29T10:47:13.135 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.152+0000 7f18ba7fc640 10 client.4634.objecter ms_handle_connect 0x7f18a806df90 2024-03-29T10:47:13.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.162+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18bc050420 con 0x7f18c40b38c0 2024-03-29T10:47:13.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.163+0000 7f18ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.145 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.163+0000 7f18ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:13.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.488+0000 7f18ca934640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.488+0000 7f18ca934640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.488+0000 7f18ca934640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:13.471 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.488+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18ac005330 con 0x7f18c40b38c0 2024-03-29T10:47:13.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.490+0000 7f18ba7fc640 1 -- 172.21.15.151:0/3010716554 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7f18bc050420 con 0x7f18c40b38c0 2024-03-29T10:47:13.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.490+0000 7f18ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.472 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.490+0000 7f18ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:13.473 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:13.473 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:13.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.493+0000 7f18ca934640 10 librados: watch_flush enter 2024-03-29T10:47:13.476 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 10 librados: watch_flush exit 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 10 client.4634.objecter successfully canceled tick 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 >> v1:172.21.15.151:6816/1549873000 conn(0x7f18a806df90 legacy=0x7f18a8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 10 monclient: shutdown 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 >> v1:172.21.15.151:6790/0 conn(0x7f18c40b38c0 legacy=0x7f18c40b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 shutdown_connections 2024-03-29T10:47:13.477 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.494+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 >> 172.21.15.151:0/3010716554 conn(0x7f18c4066a00 msgr2=0x7f18c4145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:13.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.497+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 shutdown_connections 2024-03-29T10:47:13.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.497+0000 7f18ca934640 1 -- 172.21.15.151:0/3010716554 wait complete. 2024-03-29T10:47:13.480 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:13.497+0000 7f18ca934640 1 librados: shutdown 2024-03-29T10:47:13.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:20.307 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:20.447 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.464+0000 7f4d965c3640 10 monclient: get_monmap_and_config 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.464+0000 7f4d965c3640 10 monclient: build_initial_monmap 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.464+0000 7f4d965c3640 10 monclient: monmap: 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:20.466379+0000 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:20.466379+0000 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:20.448 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.449 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.465+0000 7f4d965c3640 1 Processor -- start 2024-03-29T10:47:20.449 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.465+0000 7f4d965c3640 1 -- start start 2024-03-29T10:47:20.449 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.465+0000 7f4d965c3640 10 monclient: init 2024-03-29T10:47:20.455 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:20.455 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:20.455 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880c4e90 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:20.455 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880cb1d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d880d1140 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880c7830 con 0x7f4d880cb1d0 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d88008de0 con 0x7f4d880d1140 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8802ffd0 con 0x7f4d880c4e90 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:20.456 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.466+0000 7f4d965c3640 10 monclient(hunting): authenticate will time out at 1571.637412s 2024-03-29T10:47:20.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.467+0000 7f4d94dc0640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4d880cb1d0 0x7f4d880cb620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:48636/0 (socket says 172.21.15.151:48636) 2024-03-29T10:47:20.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.467+0000 7f4d94dc0640 1 -- 172.21.15.151:0/1877836218 learned_addr learned my addr 172.21.15.151:0/1877836218 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:20.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633054081 0 0) 0x7f4d88008de0 con 0x7f4d880d1140 2024-03-29T10:47:20.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.457 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 10 monclient(hunting): my global_id is 4570 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c003520 con 0x7f4d880d1140 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.468+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211417250 0 0) 0x7f4d8802ffd0 con 0x7f4d880c4e90 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.458 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d88008de0 con 0x7f4d880c4e90 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635707839 0 0) 0x7f4d880c7830 con 0x7f4d880cb1d0 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.459 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:47:20.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8802ffd0 con 0x7f4d880cb1d0 2024-03-29T10:47:20.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216478375 0 0) 0x7f4d7c003520 con 0x7f4d880d1140 2024-03-29T10:47:20.460 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 >> v1:172.21.15.151:6790/0 conn(0x7f4d880c4e90 legacy=0x7f4d88006ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 >> v1:172.21.15.151:6789/0 conn(0x7f4d880cb1d0 legacy=0x7f4d880cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:47:20.461 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880302b0 con 0x7f4d880d1140 2024-03-29T10:47:20.462 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:20.462 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:20.462 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:20.462 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d8c001610 con 0x7f4d880d1140 2024-03-29T10:47:20.462 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.469+0000 7f4d87fff640 10 monclient: dump: 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.463 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.464 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d965c3640 5 monclient: authenticate success, global_id 4570 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d965c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d8c001770 con 0x7f4d880d1140 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 1 -- 172.21.15.151:0/1877836218 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d8c001bc0 con 0x7f4d880d1140 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.465 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d87fff640 10 monclient: dump: 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.466 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d965c3640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.470+0000 7f4d965c3640 4 set_mon_vals no callback set 2024-03-29T10:47:20.467 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.471+0000 7f4d965c3640 10 monclient: shutdown 2024-03-29T10:47:20.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.471+0000 7f4d965c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:20.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.471+0000 7f4d965c3640 1 -- 172.21.15.151:0/1877836218 >> v1:172.21.15.169:6789/0 conn(0x7f4d880d1140 legacy=0x7f4d880d3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.472+0000 7f4d965c3640 1 -- 172.21.15.151:0/1877836218 shutdown_connections 2024-03-29T10:47:20.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.472+0000 7f4d965c3640 1 -- 172.21.15.151:0/1877836218 >> 172.21.15.151:0/1877836218 conn(0x7f4d88005c60 msgr2=0x7f4d88006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:20.468 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.472+0000 7f4d965c3640 1 -- 172.21.15.151:0/1877836218 shutdown_connections 2024-03-29T10:47:20.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.472+0000 7f4d965c3640 1 -- 172.21.15.151:0/1877836218 wait complete. 2024-03-29T10:47:20.482 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 10 monclient: build_initial_monmap 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 10 monclient: monmap: 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:20.483 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 librados: starting msgr at 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 librados: starting objecter 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 Processor -- start 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 -- start start 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 librados: setting wanted keys 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 1 librados: calling monclient init 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.473+0000 7f4d965c3640 10 monclient: init 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.474+0000 7f4d965c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.474+0000 7f4d965c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:20.484 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.474+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880c4e90 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.474+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880cb1d0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d94dc0640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4d880cb1d0 0x7f4d8802afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:56000/0 (socket says 172.21.15.151:56000) 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d880d1140 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8802e2b0 con 0x7f4d880cb1d0 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8802ce70 con 0x7f4d880c4e90 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d94dc0640 1 -- 172.21.15.151:0/1541108652 learned_addr learned my addr 172.21.15.151:0/1541108652 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8802d150 con 0x7f4d880d1140 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d965c3640 10 monclient(hunting): authenticate will time out at 1571.645805s 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d880d1140 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d880cb1d0 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:20.485 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070532984 0 0) 0x7f4d8802d150 con 0x7f4d880d1140 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74004eb0 con 0x7f4d880d1140 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d880c4e90 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677863539 0 0) 0x7f4d8802e2b0 con 0x7f4d880cb1d0 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:47:20.486 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.475+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8802d150 con 0x7f4d880cb1d0 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616758800 0 0) 0x7f4d8802ce70 con 0x7f4d880c4e90 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4573 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8802e2b0 con 0x7f4d880c4e90 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454570813 0 0) 0x7f4d74004eb0 con 0x7f4d880d1140 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8802ce70 con 0x7f4d880d1140 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d8c001a80 con 0x7f4d880d1140 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.487 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589968275 0 0) 0x7f4d8802d150 con 0x7f4d880cb1d0 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d74004eb0 con 0x7f4d880cb1d0 2024-03-29T10:47:20.488 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d9004fdd0 con 0x7f4d880cb1d0 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.476+0000 7f4d85ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.489 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863574723 0 0) 0x7f4d8802ce70 con 0x7f4d880d1140 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 >> v1:172.21.15.169:6789/0 conn(0x7f4d880c4e90 legacy=0x7f4d8802dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 >> v1:172.21.15.151:6789/0 conn(0x7f4d880cb1d0 legacy=0x7f4d8802afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 1 monclient: found mon.c 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8802e4e0 con 0x7f4d880d1140 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 5 monclient: authenticate success, global_id 4640 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 10 monclient: _renew_subs 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:20.490 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8802c080 con 0x7f4d880d1140 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 10 monclient: _renew_subs 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8816cf10 con 0x7f4d880d1140 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.477+0000 7f4d965c3640 1 librados: init done 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d8c000fe0 con 0x7f4d880d1140 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f4d8c0038a0 con 0x7f4d880d1140 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d85ffb640 10 monclient: dump: 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.491 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:20.492 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.478+0000 7f4d86ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.479+0000 7f4d965c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.480+0000 7f4d965c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.480+0000 7f4d965c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.480+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70005080 con 0x7f4d880d1140 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.480+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f4d8c019870 con 0x7f4d880d1140 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7f4d8c083720 con 0x7f4d880d1140 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 10 client.4640.objecter ms_dispatch 0x7f4d90000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d955c1640 10 monclient: get_auth_request con 0x7f4d7406de70 auth_method 0 2024-03-29T10:47:20.493 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.481+0000 7f4d85ffb640 10 client.4640.objecter ms_handle_connect 0x7f4d7406de70 2024-03-29T10:47:20.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.488+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d8c04ee30 con 0x7f4d880d1140 2024-03-29T10:47:20.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.488+0000 7f4d85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.494 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.488+0000 7f4d85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:20.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.821+0000 7f4d965c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.821+0000 7f4d965c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.821+0000 7f4d965c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.151:6790/0 2024-03-29T10:47:20.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.821+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 --> v1:172.21.15.151:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d70005330 con 0x7f4d880d1140 2024-03-29T10:47:20.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.823+0000 7f4d85ffb640 1 -- 172.21.15.151:0/1541108652 <== mon.2 v1:172.21.15.151:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7f4d8c050f30 con 0x7f4d880d1140 2024-03-29T10:47:20.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.823+0000 7f4d85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.823+0000 7f4d85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:20.807 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:20.807 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:20.810 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.827+0000 7f4d965c3640 10 librados: watch_flush enter 2024-03-29T10:47:20.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.829+0000 7f4d965c3640 10 librados: watch_flush exit 2024-03-29T10:47:20.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.829+0000 7f4d965c3640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T10:47:20.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.829+0000 7f4d965c3640 10 client.4640.objecter successfully canceled tick 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 >> v1:172.21.15.151:6816/1549873000 conn(0x7f4d7406de70 legacy=0x7f4d74070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 10 monclient: shutdown 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 >> v1:172.21.15.151:6790/0 conn(0x7f4d880d1140 legacy=0x7f4d8802b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 shutdown_connections 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 >> 172.21.15.151:0/1541108652 conn(0x7f4d88005c60 msgr2=0x7f4d880c1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:20.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.830+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 shutdown_connections 2024-03-29T10:47:20.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.831+0000 7f4d965c3640 1 -- 172.21.15.151:0/1541108652 wait complete. 2024-03-29T10:47:20.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:20.831+0000 7f4d965c3640 1 librados: shutdown 2024-03-29T10:47:20.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:27.640 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:27.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.818+0000 7fb87d086640 10 monclient: get_monmap_and_config 2024-03-29T10:47:27.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.818+0000 7fb87d086640 10 monclient: build_initial_monmap 2024-03-29T10:47:27.800 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.818+0000 7fb87d086640 10 monclient: monmap: 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:27.819035+0000 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:27.819035+0000 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.801 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.818+0000 7fb87d086640 1 Processor -- start 2024-03-29T10:47:27.805 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 1 -- start start 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 10 monclient: init 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-a con 0x7fb8780b0770 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.819+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-c con 0x7fb87814a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-b con 0x7fb8780ac5a0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780aba70 con 0x7fb8780b0770 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780aecb0 con 0x7fb87814a700 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780af150 con 0x7fb8780ac5a0 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87d086640 10 monclient(hunting): authenticate will time out at 1578.992795s 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87759e640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb8780ac5a0 0x7fb878151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:40441/0 (socket says 172.21.15.151:40441) 2024-03-29T10:47:27.806 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.823+0000 7fb87759e640 1 -- 172.21.15.151:0/1525197624 learned_addr learned my addr 172.21.15.151:0/1525197624 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117292413 0 0) 0x7fb8780af150 con 0x7fb8780ac5a0 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.825+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003520 con 0x7fb8780ac5a0 2024-03-29T10:47:27.808 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.826+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1533937436 0 0) 0x7fb860003520 con 0x7fb8780ac5a0 2024-03-29T10:47:27.809 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.826+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 >> v1:172.21.15.169:6789/0 conn(0x7fb87814a700 legacy=0x7fb8780abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 >> v1:172.21.15.151:6789/0 conn(0x7fb8780b0770 legacy=0x7fb878149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 1 monclient: found mon.noname-b 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780ac9d0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.812 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: _finish_auth 0 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb87d086640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb87d086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb870001a10 con 0x7fb8780ac5a0 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb875d9b640 10 monclient: dump: 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.813 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.827+0000 7fb87d086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.828+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001ca0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.828+0000 7fb875d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.828+0000 7fb87d086640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.828+0000 7fb87d086640 4 set_mon_vals no callback set 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb875d9b640 1 -- 172.21.15.151:0/1525197624 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb870002030 con 0x7fb8780ac5a0 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb875d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb875d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb875d9b640 10 monclient: dump: 2024-03-29T10:47:27.814 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb87d086640 10 monclient: shutdown 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb87d086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb87d086640 1 -- 172.21.15.151:0/1525197624 >> v1:172.21.15.151:6790/0 conn(0x7fb8780ac5a0 legacy=0x7fb878151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb87d086640 1 -- 172.21.15.151:0/1525197624 shutdown_connections 2024-03-29T10:47:27.815 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.829+0000 7fb87d086640 1 -- 172.21.15.151:0/1525197624 >> 172.21.15.151:0/1525197624 conn(0x7fb87806b660 msgr2=0x7fb87806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:27.816 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.830+0000 7fb87d086640 1 -- 172.21.15.151:0/1525197624 shutdown_connections 2024-03-29T10:47:27.816 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.830+0000 7fb87d086640 1 -- 172.21.15.151:0/1525197624 wait complete. 2024-03-29T10:47:27.816 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.831+0000 7fb87d086640 10 monclient: build_initial_monmap 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.831+0000 7fb87d086640 10 monclient: monmap: 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 librados: starting msgr at 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 librados: starting objecter 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 Processor -- start 2024-03-29T10:47:27.817 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 -- start start 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 librados: setting wanted keys 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 1 librados: calling monclient init 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.832+0000 7fb87d086640 10 monclient: init 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.833+0000 7fb87d086640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.833+0000 7fb87d086640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.833+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-a con 0x7fb8780ac5a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:27.818 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.833+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-c con 0x7fb8780b0770 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 10 monclient(hunting): picked mon.noname-b con 0x7fb87814a700 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878124000 con 0x7fb8780ac5a0 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878070130 con 0x7fb87814a700 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8781dbec0 con 0x7fb8780b0770 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:27.820 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.837+0000 7fb87d086640 10 monclient(hunting): authenticate will time out at 1579.007069s 2024-03-29T10:47:27.822 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.840+0000 7fb876d9d640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb8780ac5a0 0x7fb878140b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:59357/0 (socket says 172.21.15.151:59357) 2024-03-29T10:47:27.822 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.840+0000 7fb876d9d640 1 -- 172.21.15.151:0/1154414029 learned_addr learned my addr 172.21.15.151:0/1154414029 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:27.828 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8780ac5a0 2024-03-29T10:47:27.828 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:27.828 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8780b0770 2024-03-29T10:47:27.828 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410703553 0 0) 0x7fb878124000 con 0x7fb8780ac5a0 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.842+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864004eb0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1349366328 0 0) 0x7fb864004eb0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb878124000 con 0x7fb8780ac5a0 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb86c004080 con 0x7fb8780ac5a0 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:27.829 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.843+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248816452 0 0) 0x7fb8781dbec0 con 0x7fb8780b0770 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.830 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864004eb0 con 0x7fb8780b0770 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2516114705 0 0) 0x7fb878124000 con 0x7fb8780ac5a0 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 >> v1:172.21.15.151:6790/0 conn(0x7fb8780b0770 legacy=0x7fb878070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 >> v1:172.21.15.169:6789/0 conn(0x7fb87814a700 legacy=0x7fb878070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 monclient: found mon.a 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8781dc050 con 0x7fb8780ac5a0 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb85f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:27.831 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 5 monclient: authenticate success, global_id 4554 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 10 monclient: _renew_subs 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb878141260 con 0x7fb8780ac5a0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 10 monclient: _renew_subs 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb878141920 con 0x7fb8780ac5a0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.844+0000 7fb87d086640 1 librados: init done 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb86c000ed0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb874d99640 4 set_mon_vals no callback set 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb86c0012a0 con 0x7fb8780ac5a0 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.832 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.845+0000 7fb85f7fe640 10 monclient: dump: 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:27.833 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.846+0000 7fb87d086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.834 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.846+0000 7fb87d086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.847+0000 7fb87d086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.847+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb858005080 con 0x7fb8780ac5a0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.848+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb86c001400 con 0x7fb8780ac5a0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7fb86c084a50 con 0x7fb8780ac5a0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb85f7fe640 10 client.4554.objecter ms_dispatch 0x7fb878000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb85f7fe640 3 client.4554.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb85f7fe640 3 client.4554.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb85f7fe640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.849+0000 7fb87659c640 10 monclient: get_auth_request con 0x7fb86406dd70 auth_method 0 2024-03-29T10:47:27.835 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.850+0000 7fb85f7fe640 10 client.4554.objecter ms_handle_connect 0x7fb86406dd70 2024-03-29T10:47:27.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.855+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb86c050330 con 0x7fb8780ac5a0 2024-03-29T10:47:27.837 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.855+0000 7fb85f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.838 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:27.855+0000 7fb85f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:28.195 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.213+0000 7fb87d086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.196 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.213+0000 7fb87d086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.196 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.213+0000 7fb87d086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:28.196 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.213+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb858005330 con 0x7fb8780ac5a0 2024-03-29T10:47:28.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.215+0000 7fb85f7fe640 1 -- 172.21.15.151:0/1154414029 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7fb86c052260 con 0x7fb8780ac5a0 2024-03-29T10:47:28.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.215+0000 7fb85f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.215+0000 7fb85f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:28.198 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:28.198 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:28.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.218+0000 7fb87d086640 10 librados: watch_flush enter 2024-03-29T10:47:28.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 10 librados: watch_flush exit 2024-03-29T10:47:28.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T10:47:28.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 10 client.4554.objecter successfully canceled tick 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 >> v1:172.21.15.151:6816/1549873000 conn(0x7fb86406dd70 legacy=0x7fb864070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 10 monclient: shutdown 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 >> v1:172.21.15.151:6789/0 conn(0x7fb8780ac5a0 legacy=0x7fb878140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 shutdown_connections 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 >> 172.21.15.151:0/1154414029 conn(0x7fb87806b660 msgr2=0x7fb87806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 shutdown_connections 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.219+0000 7fb87d086640 1 -- 172.21.15.151:0/1154414029 wait complete. 2024-03-29T10:47:28.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:28.220+0000 7fb87d086640 1 librados: shutdown 2024-03-29T10:47:28.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:35.024 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:35.196 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.213+0000 7fd35b94b640 10 monclient: get_monmap_and_config 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.213+0000 7fd35b94b640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.214+0000 7fd35b94b640 10 monclient: monmap: 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:35.214917+0000 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:35.214917+0000 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.197 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.215+0000 7fd35b94b640 1 Processor -- start 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.215+0000 7fd35b94b640 1 -- start start 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.215+0000 7fd35b94b640 10 monclient: init 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.216+0000 7fd35b94b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.198 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.216+0000 7fd35b94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.216+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-c con 0x7fd3540b02e0 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-a con 0x7fd3540aeb90 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-b con 0x7fd3540aeff0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd3596e7640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd3540b02e0 0x7fd35413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58695/0 (socket says 172.21.15.151:58695) 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd3596e7640 1 -- 172.21.15.151:0/990664626 learned_addr learned my addr 172.21.15.151:0/990664626 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:35.201 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35413fb10 con 0x7fd3540aeb90 2024-03-29T10:47:35.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3540ae280 con 0x7fd3540b02e0 2024-03-29T10:47:35.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3541421e0 con 0x7fd3540aeff0 2024-03-29T10:47:35.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.202 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd35b94b640 10 monclient(hunting): authenticate will time out at 1586.389049s 2024-03-29T10:47:35.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153811708 0 0) 0x7fd3540ae280 con 0x7fd3540b02e0 2024-03-29T10:47:35.204 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 10 monclient(hunting): my global_id is 4576 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.219+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003520 con 0x7fd3540b02e0 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.220+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934833651 0 0) 0x7fd340003520 con 0x7fd3540b02e0 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.220+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 >> v1:172.21.15.151:6790/0 conn(0x7fd3540aeff0 legacy=0x7fd35414da00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:35.205 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.220+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 >> v1:172.21.15.151:6789/0 conn(0x7fd3540aeb90 legacy=0x7fd354142700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:35.206 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 1 monclient: found mon.noname-c 2024-03-29T10:47:35.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354145660 con 0x7fd3540b02e0 2024-03-29T10:47:35.207 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd34c001a10 con 0x7fd3540b02e0 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: dump: 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:35.208 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:35.209 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd35b94b640 5 monclient: authenticate success, global_id 4576 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd35b94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34c001be0 con 0x7fd3540b02e0 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 1 -- 172.21.15.151:0/990664626 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd34c002030 con 0x7fd3540b02e0 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.210 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd34bfff640 10 monclient: dump: 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.211 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd35b94b640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.223+0000 7fd35b94b640 4 set_mon_vals no callback set 2024-03-29T10:47:35.212 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.224+0000 7fd35b94b640 10 monclient: shutdown 2024-03-29T10:47:35.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.224+0000 7fd35b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:35.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.224+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 >> v1:172.21.15.169:6789/0 conn(0x7fd3540b02e0 legacy=0x7fd35413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.227+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 shutdown_connections 2024-03-29T10:47:35.213 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.227+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 >> 172.21.15.151:0/990664626 conn(0x7fd35406b660 msgr2=0x7fd35406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:35.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.230+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 shutdown_connections 2024-03-29T10:47:35.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.231+0000 7fd35b94b640 1 -- 172.21.15.151:0/990664626 wait complete. 2024-03-29T10:47:35.216 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.233+0000 7fd35b94b640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.233+0000 7fd35b94b640 10 monclient: monmap: 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.233+0000 7fd35b94b640 1 librados: starting msgr at 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.233+0000 7fd35b94b640 1 librados: starting objecter 2024-03-29T10:47:35.217 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.234+0000 7fd35b94b640 1 Processor -- start 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.234+0000 7fd35b94b640 1 -- start start 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.234+0000 7fd35b94b640 1 librados: setting wanted keys 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.234+0000 7fd35b94b640 1 librados: calling monclient init 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.234+0000 7fd35b94b640 10 monclient: init 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.218 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:35.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-b con 0x7fd3540aeb90 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:35.220 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-c con 0x7fd3540aeff0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient(hunting): picked mon.noname-a con 0x7fd3540b02e0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354143710 con 0x7fd3540b02e0 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354145390 con 0x7fd3540aeb90 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd354143d00 con 0x7fd3540aeff0 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.235+0000 7fd35b94b640 10 monclient(hunting): authenticate will time out at 1586.404997s 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.238+0000 7fd3596e7640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd3540aeb90 0x7fd35412d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.151:58833/0 (socket says 172.21.15.151:58833) 2024-03-29T10:47:35.221 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.238+0000 7fd3596e7640 1 -- 172.21.15.151:0/821026288 learned_addr learned my addr 172.21.15.151:0/821026288 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.240+0000 7fd349ffb640 10 client.?.objecter ms_handle_connect 0x7fd3540aeb90 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.240+0000 7fd349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962020062 0 0) 0x7fd354145390 con 0x7fd3540aeb90 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.229 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 10 monclient(hunting): my global_id is 4579 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.241+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338004eb0 con 0x7fd3540aeb90 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200574523 0 0) 0x7fd338004eb0 con 0x7fd3540aeb90 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd354145390 con 0x7fd3540aeb90 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd34c001fb0 con 0x7fd3540aeb90 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.230 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 client.?.objecter ms_handle_connect 0x7fd3540b02e0 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 client.?.objecter ms_handle_connect 0x7fd3540aeff0 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.231 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1731535534 0 0) 0x7fd354145390 con 0x7fd3540aeb90 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 >> v1:172.21.15.151:6790/0 conn(0x7fd3540aeff0 legacy=0x7fd3541da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 >> v1:172.21.15.151:6789/0 conn(0x7fd3540b02e0 legacy=0x7fd3541e2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 monclient: found mon.b 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354143e90 con 0x7fd3540aeb90 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.242+0000 7fd349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 5 monclient: authenticate success, global_id 4579 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 10 monclient: _renew_subs 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.232 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35412de40 con 0x7fd3540aeb90 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 10 monclient: _renew_subs 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3540ad340 con 0x7fd3540aeb90 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd35b94b640 1 librados: init done 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34c004e80 con 0x7fd3540aeb90 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fd34c005290 con 0x7fd3540aeb90 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.243+0000 7fd349ffb640 10 monclient: dump: 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:35.233 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.244+0000 7fd34b7fe640 4 set_mon_vals no callback set 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.245+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fd34c0053f0 con 0x7fd3540aeb90 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7fd34c084f60 con 0x7fd3540aeb90 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd358ee6640 10 monclient: get_auth_request con 0x7fd33806dd90 auth_method 0 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd35b94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd35b94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd35b94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.247+0000 7fd349ffb640 10 client.4579.objecter ms_dispatch 0x7fd354000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.246+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd334005080 con 0x7fd3540aeb90 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.247+0000 7fd349ffb640 3 client.4579.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.247+0000 7fd349ffb640 3 client.4579.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.247+0000 7fd349ffb640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-29T10:47:35.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.247+0000 7fd349ffb640 10 client.4579.objecter ms_handle_connect 0x7fd33806dd90 2024-03-29T10:47:35.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.252+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd34c050840 con 0x7fd3540aeb90 2024-03-29T10:47:35.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.252+0000 7fd349ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.236 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.252+0000 7fd349ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:35.565 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:35.565 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:35.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.579+0000 7fd35b94b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.579+0000 7fd35b94b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.579+0000 7fd35b94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.579+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd334005330 con 0x7fd3540aeb90 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.581+0000 7fd349ffb640 1 -- 172.21.15.151:0/821026288 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7fd34c052770 con 0x7fd3540aeb90 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.581+0000 7fd349ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.581+0000 7fd349ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:35.568 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.586+0000 7fd35b94b640 10 librados: watch_flush enter 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 10 librados: watch_flush exit 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 10 client.4579.objecter successfully canceled tick 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 >> v1:172.21.15.151:6816/1549873000 conn(0x7fd33806dd90 legacy=0x7fd338070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 10 monclient: shutdown 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.587+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 >> v1:172.21.15.169:6789/0 conn(0x7fd3540aeb90 legacy=0x7fd35412d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.588+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 shutdown_connections 2024-03-29T10:47:35.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.588+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 >> 172.21.15.151:0/821026288 conn(0x7fd35406b660 msgr2=0x7fd3540b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:35.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.588+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 shutdown_connections 2024-03-29T10:47:35.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.588+0000 7fd35b94b640 1 -- 172.21.15.151:0/821026288 wait complete. 2024-03-29T10:47:35.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:35.588+0000 7fd35b94b640 1 librados: shutdown 2024-03-29T10:47:35.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:42.396 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.557+0000 7f6d6cd6b640 10 monclient: get_monmap_and_config 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.557+0000 7f6d6cd6b640 10 monclient: build_initial_monmap 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.557+0000 7f6d6cd6b640 10 monclient: monmap: 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:42.558326+0000 2024-03-29T10:47:42.540 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:42.558326+0000 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:42.541 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.559+0000 7f6d6cd6b640 1 Processor -- start 2024-03-29T10:47:42.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.559+0000 7f6d6cd6b640 1 -- start start 2024-03-29T10:47:42.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.559+0000 7f6d6cd6b640 10 monclient: init 2024-03-29T10:47:42.542 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.560+0000 7f6d6cd6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:42.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.560+0000 7f6d6cd6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:42.543 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.560+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f6d680a9ac0 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:42.547 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.560+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6813ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f6d680af900 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680afd10 con 0x7f6d6813ee90 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680af6f0 con 0x7f6d680af900 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 1 -- --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680aac80 con 0x7f6d680a9ac0 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6cd6b640 10 monclient(hunting): authenticate will time out at 1593.733446s 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.563+0000 7f6d6659c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f6d680a9ac0 0x7f6d680adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:55116/0 (socket says 172.21.15.151:55116) 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d6659c640 1 -- 172.21.15.151:0/2304807144 learned_addr learned my addr 172.21.15.151:0/2304807144 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125400704 0 0) 0x7f6d680aac80 con 0x7f6d680a9ac0 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.548 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d54003520 con 0x7f6d680a9ac0 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.564+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477971413 0 0) 0x7f6d680afd10 con 0x7f6d6813ee90 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d680aac80 con 0x7f6d6813ee90 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624178904 0 0) 0x7f6d54003520 con 0x7f6d680a9ac0 2024-03-29T10:47:42.549 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.565+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 >> v1:172.21.15.169:6789/0 conn(0x7f6d680af900 legacy=0x7f6d68149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 >> v1:172.21.15.151:6789/0 conn(0x7f6d6813ee90 legacy=0x7f6d6813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 monclient: found mon.noname-b 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.151:6790/0 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 --> v1:172.21.15.151:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680aaf60 con 0x7f6d680a9ac0 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: _finish_auth 0 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d58001690 con 0x7f6d680a9ac0 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:42.552 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: dump: 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d6cd6b640 5 monclient: authenticate success, global_id 4649 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d6cd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.2 v1:172.21.15.151:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d580017f0 con 0x7f6d680a9ac0 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:42.553 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 1 -- 172.21.15.151:0/2304807144 <== mon.2 v1:172.21.15.151:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d58001c40 con 0x7f6d680a9ac0 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d65d9b640 10 monclient: dump: 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:42.554 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d6cd6b640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.567+0000 7f6d6cd6b640 4 set_mon_vals no callback set 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 10 monclient: shutdown 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/2304807144 >> v1:172.21.15.151:6790/0 conn(0x7f6d680a9ac0 legacy=0x7f6d680adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/2304807144 shutdown_connections 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/2304807144 >> 172.21.15.151:0/2304807144 conn(0x7f6d680b35e0 msgr2=0x7f6d680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.568+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/2304807144 shutdown_connections 2024-03-29T10:47:42.555 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.569+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/2304807144 wait complete. 2024-03-29T10:47:42.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.570+0000 7f6d6cd6b640 10 monclient: build_initial_monmap 2024-03-29T10:47:42.557 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.570+0000 7f6d6cd6b640 10 monclient: monmap: 2024-03-29T10:47:42.557 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:42.557 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.571+0000 7f6d6cd6b640 1 librados: starting msgr at 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.571+0000 7f6d6cd6b640 1 librados: starting objecter 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.571+0000 7f6d6cd6b640 1 Processor -- start 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.572+0000 7f6d6cd6b640 1 -- start start 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.572+0000 7f6d6cd6b640 1 librados: setting wanted keys 2024-03-29T10:47:42.558 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.572+0000 7f6d6cd6b640 1 librados: calling monclient init 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.572+0000 7f6d6cd6b640 10 monclient: init 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6cd6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6cd6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6659c640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7f6d680af900 0x7f6d68141ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:56250/0 (socket says 172.21.15.151:56250) 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6659c640 1 -- 172.21.15.151:0/3045068193 learned_addr learned my addr 172.21.15.151:0/3045068193 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d5effd640 10 client.?.objecter ms_handle_connect 0x7f6d680af900 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f6d680af900 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.573+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6813ee90 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f6d681d9700 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:42.559 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6806fb00 con 0x7f6d6813ee90 2024-03-29T10:47:42.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6813f5c0 con 0x7f6d681d9700 2024-03-29T10:47:42.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6806fd70 con 0x7f6d680af900 2024-03-29T10:47:42.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663999934 0 0) 0x7f6d6806fd70 con 0x7f6d680af900 2024-03-29T10:47:42.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:42.560 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.574+0000 7f6d6cd6b640 10 monclient(hunting): authenticate will time out at 1593.743998s 2024-03-29T10:47:42.561 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.576+0000 7f6d5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.561 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.576+0000 7f6d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.561 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.576+0000 7f6d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.561 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.576+0000 7f6d5effd640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:47:42.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.576+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d48004eb0 con 0x7f6d680af900 2024-03-29T10:47:42.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.577+0000 7f6d5effd640 10 client.?.objecter ms_handle_connect 0x7f6d681d9700 2024-03-29T10:47:42.562 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.577+0000 7f6d5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 602429094 0 0) 0x7f6d48004eb0 con 0x7f6d680af900 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6806fd70 con 0x7f6d680af900 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d58001b00 con 0x7f6d680af900 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): dump: 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:42.564 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528991915 0 0) 0x7f6d6813f5c0 con 0x7f6d681d9700 2024-03-29T10:47:42.565 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 10 monclient(hunting): my global_id is 4582 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.580+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d48004eb0 con 0x7f6d681d9700 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 client.?.objecter ms_handle_connect 0x7f6d6813ee90 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640115003 0 0) 0x7f6d6806fb00 con 0x7f6d6813ee90 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.566 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6813f5c0 con 0x7f6d6813ee90 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199396732 0 0) 0x7f6d6813f5c0 con 0x7f6d6813ee90 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6806fb00 con 0x7f6d6813ee90 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d50001780 con 0x7f6d6813ee90 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.581+0000 7f6d5effd640 10 monclient(hunting): dump: 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.567 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:42.568 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2449783640 0 0) 0x7f6d6806fb00 con 0x7f6d6813ee90 2024-03-29T10:47:42.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 >> v1:172.21.15.151:6790/0 conn(0x7f6d680af900 legacy=0x7f6d68141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.569 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 >> v1:172.21.15.169:6789/0 conn(0x7f6d681d9700 legacy=0x7f6d681e00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 1 monclient: found mon.a 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6806ea10 con 0x7f6d6813ee90 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 10 monclient: _finish_auth 0 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.582+0000 7f6d5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:47:42.570 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 10 monclient: _renew_subs 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6806ec60 con 0x7f6d6813ee90 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 10 monclient: _renew_subs 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d681e19e0 con 0x7f6d6813ee90 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.583+0000 7f6d6cd6b640 1 librados: init done 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.584+0000 7f6d6cd6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.584+0000 7f6d6cd6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.584+0000 7f6d6cd6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.584+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6814d9f0 con 0x7f6d6813ee90 2024-03-29T10:47:42.571 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d50001c30 con 0x7f6d6813ee90 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7f6d50003790 con 0x7f6d6813ee90 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d5effd640 10 monclient: dump: 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.572 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:42.573 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:42.573 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:42.573 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:42.573 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:42.573 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.586+0000 7f6d64d99640 4 set_mon_vals no callback set 2024-03-29T10:47:42.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.588+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7f6d50019760 con 0x7f6d6813ee90 2024-03-29T10:47:42.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.590+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7f6d50083630 con 0x7f6d6813ee90 2024-03-29T10:47:42.574 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.590+0000 7f6d5effd640 10 client.4560.objecter ms_dispatch 0x7f6d68000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:42.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.590+0000 7f6d5effd640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:42.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.590+0000 7f6d5effd640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:42.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.590+0000 7f6d5effd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T10:47:42.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.591+0000 7f6d6659c640 10 monclient: get_auth_request con 0x7f6d4806dfb0 auth_method 0 2024-03-29T10:47:42.575 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.592+0000 7f6d5effd640 10 client.4560.objecter ms_handle_connect 0x7f6d4806dfb0 2024-03-29T10:47:42.578 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.596+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d5004ef10 con 0x7f6d6813ee90 2024-03-29T10:47:42.579 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.596+0000 7f6d5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.579 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.596+0000 7f6d5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:42.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.920+0000 7f6d6cd6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.920+0000 7f6d6cd6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.920+0000 7f6d6cd6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:42.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.920+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d6813f3b0 con 0x7f6d6813ee90 2024-03-29T10:47:42.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.922+0000 7f6d5effd640 1 -- 172.21.15.151:0/3045068193 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7f6d50050e40 con 0x7f6d6813ee90 2024-03-29T10:47:42.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.922+0000 7f6d5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.922+0000 7f6d5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:42.905 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:42.905 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:42.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 10 librados: watch_flush enter 2024-03-29T10:47:42.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 10 librados: watch_flush exit 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 10 client.4560.objecter successfully canceled tick 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 >> v1:172.21.15.151:6816/1549873000 conn(0x7f6d4806dfb0 legacy=0x7f6d48070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 10 monclient: shutdown 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:42.910 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.926+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 >> v1:172.21.15.151:6789/0 conn(0x7f6d6813ee90 legacy=0x7f6d681d8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.927+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 shutdown_connections 2024-03-29T10:47:42.911 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.927+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 >> 172.21.15.151:0/3045068193 conn(0x7f6d680b35e0 msgr2=0x7f6d680ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:42.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.928+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 shutdown_connections 2024-03-29T10:47:42.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.928+0000 7f6d6cd6b640 1 -- 172.21.15.151:0/3045068193 wait complete. 2024-03-29T10:47:42.912 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:42.928+0000 7f6d6cd6b640 1 librados: shutdown 2024-03-29T10:47:42.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:49.735 DEBUG:teuthology.orchestra.run.smithi151:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.909+0000 7fb560c38640 10 monclient: get_monmap_and_config 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.909+0000 7fb560c38640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.909+0000 7fb560c38640 10 monclient: monmap: 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:47:49.910801+0000 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:47:49.910801+0000 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:49.892 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-29T10:47:49.893 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-b 2024-03-29T10:47:49.893 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.893 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.910+0000 7fb560c38640 1 Processor -- start 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.911+0000 7fb560c38640 1 -- start start 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.911+0000 7fb560c38640 10 monclient: init 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.912+0000 7fb560c38640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.912+0000 7fb560c38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.912+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-c con 0x7fb55c0b4920 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:49.894 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-b con 0x7fb55c0b4200 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:49.895 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-a con 0x7fb55c13f9a0 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:49.895 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c0b0930 con 0x7fb55c13f9a0 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb55affd640 1 --1- >> v1:172.21.15.151:6790/0 conn(0x7fb55c0b4200 0x7fb55c0b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6790/0 says I am v1:172.21.15.151:41786/0 (socket says 172.21.15.151:41786) 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb55affd640 1 -- 172.21.15.151:0/2432376689 learned_addr learned my addr 172.21.15.151:0/2432376689 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c0af460 con 0x7fb55c0b4920 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c0aeda0 con 0x7fb55c0b4200 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.896 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.913+0000 7fb560c38640 10 monclient(hunting): authenticate will time out at 1601.083275s 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427694564 0 0) 0x7fb55c0b0930 con 0x7fb55c13f9a0 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb544003520 con 0x7fb55c13f9a0 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145845242 0 0) 0x7fb55c0af460 con 0x7fb55c0b4920 2024-03-29T10:47:49.900 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.914+0000 7fb55a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient(hunting): my global_id is 4585 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c0b0930 con 0x7fb55c0b4920 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877071465 0 0) 0x7fb544003520 con 0x7fb55c13f9a0 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 >> v1:172.21.15.151:6790/0 conn(0x7fb55c0b4200 legacy=0x7fb55c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 >> v1:172.21.15.169:6789/0 conn(0x7fb55c0b4920 legacy=0x7fb55c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.151:6789/0 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0af980 con 0x7fb55c13f9a0 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.901 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb554001a10 con 0x7fb55c13f9a0 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb55a7fc640 10 monclient: dump: 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:49.902 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb560c38640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.915+0000 7fb560c38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.0 v1:172.21.15.151:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb554001c30 con 0x7fb55c13f9a0 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb560c38640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 1 -- 172.21.15.151:0/2432376689 <== mon.0 v1:172.21.15.151:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb554001f40 con 0x7fb55c13f9a0 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb560c38640 4 set_mon_vals no callback set 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.916+0000 7fb55a7fc640 10 monclient: dump: 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.903 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.917+0000 7fb560c38640 10 monclient: shutdown 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.917+0000 7fb560c38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.917+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 >> v1:172.21.15.151:6789/0 conn(0x7fb55c13f9a0 legacy=0x7fb55c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.918+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 shutdown_connections 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.918+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 >> 172.21.15.151:0/2432376689 conn(0x7fb55c06d230 msgr2=0x7fb55c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.918+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 shutdown_connections 2024-03-29T10:47:49.904 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.918+0000 7fb560c38640 1 -- 172.21.15.151:0/2432376689 wait complete. 2024-03-29T10:47:49.906 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.919+0000 7fb560c38640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.919+0000 7fb560c38640 10 monclient: monmap: 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:epoch 0 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:last_changed 0.000000 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:created 0.000000 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.noname-a 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.noname-c 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.919+0000 7fb560c38640 1 librados: starting msgr at 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.919+0000 7fb560c38640 1 librados: starting objecter 2024-03-29T10:47:49.907 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.920+0000 7fb560c38640 1 Processor -- start 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.920+0000 7fb560c38640 1 -- start start 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.920+0000 7fb560c38640 1 librados: setting wanted keys 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.920+0000 7fb560c38640 1 librados: calling monclient init 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.920+0000 7fb560c38640 10 monclient: init 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.921+0000 7fb560c38640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.921+0000 7fb560c38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.921+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-a con 0x7fb55c0b4200 addr v1:172.21.15.151:6789/0 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.921+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-c con 0x7fb55c0b4920 addr v1:172.21.15.151:6790/0 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.921+0000 7fb55b7fe640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb55c0b4200 0x7fb55c1d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:33964/0 (socket says 172.21.15.151:33964) 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.922+0000 7fb55b7fe640 1 -- 172.21.15.151:0/3178006491 learned_addr learned my addr 172.21.15.151:0/3178006491 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.922+0000 7fb543fff640 10 client.?.objecter ms_handle_connect 0x7fb55c0b4200 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.922+0000 7fb543fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.922+0000 7fb543fff640 10 client.?.objecter ms_handle_connect 0x7fb55c0b4920 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.922+0000 7fb543fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.908 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 10 monclient(hunting): picked mon.noname-b con 0x7fb55c13f9a0 addr v1:172.21.15.169:6789/0 2024-03-29T10:47:49.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c1397b0 con 0x7fb55c0b4200 2024-03-29T10:47:49.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c0ab170 con 0x7fb55c13f9a0 2024-03-29T10:47:49.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c1d71d0 con 0x7fb55c0b4920 2024-03-29T10:47:49.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.909 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.923+0000 7fb560c38640 10 monclient(hunting): authenticate will time out at 1601.093258s 2024-03-29T10:47:49.914 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646885192 0 0) 0x7fb55c1397b0 con 0x7fb55c0b4200 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb548004eb0 con 0x7fb55c0b4200 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.2 v1:172.21.15.151:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629105374 0 0) 0x7fb55c1d71d0 con 0x7fb55c0b4920 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c1397b0 con 0x7fb55c0b4920 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335974350 0 0) 0x7fb548004eb0 con 0x7fb55c0b4200 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb55c1d71d0 con 0x7fb55c0b4200 2024-03-29T10:47:49.915 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb54c001a10 con 0x7fb55c0b4200 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): dump: 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.925+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.2 v1:172.21.15.151:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690721467 0 0) 0x7fb55c1397b0 con 0x7fb55c0b4920 2024-03-29T10:47:49.916 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb548004eb0 con 0x7fb55c0b4920 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.2 v1:172.21.15.151:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb550001780 con 0x7fb55c0b4920 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient(hunting): dump: 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:49.917 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3353326384 0 0) 0x7fb55c1d71d0 con 0x7fb55c0b4200 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 >> v1:172.21.15.151:6790/0 conn(0x7fb55c0b4920 legacy=0x7fb55c138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 >> v1:172.21.15.169:6789/0 conn(0x7fb55c13f9a0 legacy=0x7fb55c1390a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 monclient: found mon.a 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0aa010 con 0x7fb55c0b4200 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb54c001c50 con 0x7fb55c0b4200 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.918 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3130148504 0 0) 0x7fb54c003a60 con 0x7fb55c0b4200 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb543fff640 10 monclient: dump: 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:epoch 1 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:fsid 87d1e7d1-c131-44d2-a9fc-ab7c13a2603e 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:last_changed 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:created 2024-03-29T10:36:14.351435+0000 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:election_strategy: 1 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:0: v1:172.21.15.151:6789/0 mon.a 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2: v1:172.21.15.151:6790/0 mon.c 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr: 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.926+0000 7fb559ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:47:49.919 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 10 monclient: _renew_subs 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c1d7e00 con 0x7fb55c0b4200 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 10 monclient: _renew_subs 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.927+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c140040 con 0x7fb55c0b4200 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.928+0000 7fb560c38640 1 librados: init done 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.929+0000 7fb560c38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.929+0000 7fb560c38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.929+0000 7fb560c38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.929+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c0aaf70 con 0x7fb55c0b4200 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.929+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2878992857 0 0) 0x7fb54c003bc0 con 0x7fb55c0b4200 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3790272500 0 0) 0x7fb54c084b60 con 0x7fb55c0b4200 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb543fff640 10 client.4566.objecter ms_dispatch 0x7fb55c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:47:49.920 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb543fff640 3 client.4566.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb543fff640 3 client.4566.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb543fff640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.930+0000 7fb55affd640 10 monclient: get_auth_request con 0x7fb54806df10 auth_method 0 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.931+0000 7fb543fff640 10 client.4566.objecter ms_handle_connect 0x7fb54806df10 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.937+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb54c050440 con 0x7fb55c0b4200 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.937+0000 7fb543fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.921 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:49.937+0000 7fb543fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb560c38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb560c38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb560c38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.151:6789/0 2024-03-29T10:47:50.234 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 --> v1:172.21.15.151:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb55c04f730 con 0x7fb55c0b4200 2024-03-29T10:47:50.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb543fff640 1 -- 172.21.15.151:0/3178006491 <== mon.0 v1:172.21.15.151:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3267627090) 0x7fb54c052370 con 0x7fb55c0b4200 2024-03-29T10:47:50.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb543fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.235 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.252+0000 7fb543fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.235 INFO:teuthology.misc.health.smithi151.stdout: 2024-03-29T10:47:50.235 INFO:teuthology.misc.health.smithi151.stdout:{"epoch":11,"fsid":"87d1e7d1-c131-44d2-a9fc-ab7c13a2603e","created":"2024-03-29T10:36:52.591244+0000","modified":"2024-03-29T10:46:57.536918+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:46:57.536918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cf044ce8-1259-4cd0-a749-9de9ff157703","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"99db8471-c2b2-48af-93dc-e4b9b1de4bd3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"26b2e1b3-eb7a-4aad-9049-de761ad4fa50","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b5ed214d-a882-467f-8676-a93556702bd4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"efc0ecc0-1f5f-484f-921a-7c52e0c0b360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"eb0fcacd-f425-4f16-91b5-7bdb850ceddc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c821dc3-03d5-446b-adc2-a299de39985f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a602a4f1-35fd-4ea9-b720-b4d97b5abaaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:50.238 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 10 librados: watch_flush enter 2024-03-29T10:47:50.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 10 librados: watch_flush exit 2024-03-29T10:47:50.288 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 10 client.4566.objecter successfully canceled tick 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 >> v1:172.21.15.151:6816/1549873000 conn(0x7fb54806df10 legacy=0x7fb5480703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 10 monclient: shutdown 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 >> v1:172.21.15.151:6789/0 conn(0x7fb55c0b4200 legacy=0x7fb55c1d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 shutdown_connections 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 >> 172.21.15.151:0/3178006491 conn(0x7fb55c06d230 msgr2=0x7fb55c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 shutdown_connections 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 -- 172.21.15.151:0/3178006491 wait complete. 2024-03-29T10:47:50.289 INFO:teuthology.misc.health.smithi151.stderr:2024-03-29T10:47:50.256+0000 7fb560c38640 1 librados: shutdown 2024-03-29T10:47:50.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:50.302 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:47:50.304 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:47:50.304 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:47:50.304 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.339 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:47:50.340 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:47:50.340 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.420 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:47:50.420 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:47:50.420 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.487 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:47:50.487 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:47:50.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.563 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:47:50.564 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:47:50.564 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.598 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:47:50.598 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:47:50.598 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.671 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:47:50.671 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:47:50.672 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.741 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:47:50.742 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:47:50.742 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.814 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:47:50.814 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:47:50.815 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:47:50.815 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.840 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:47:50.840 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:47:50.840 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.881 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:47:50.881 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:47:50.881 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:47:50.881 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.893 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:47:50.893 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:47:50.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.944 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:47:50.944 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:47:50.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:47:50.955 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:47:50.956 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:47:50.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:47:51.104 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:47:51.104 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:55.153 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:47:55.153 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:55.250 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:47:55.250 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:55.307 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:47:55.307 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:55.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:47:55.382 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:58.876 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:47:58.877 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:58.988 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:47:58.989 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:59.062 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:47:59.062 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:59.119 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:47:59.121 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/data/mon.a.tgz 2024-03-29T10:47:59.121 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-29T10:47:59.139 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.b6zP7RLwDp 2024-03-29T10:47:59.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.b6zP7RLwDp 2024-03-29T10:47:59.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.b6zP7RLwDp 2024-03-29T10:47:59.475 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.b6zP7RLwDp is 212KB 2024-03-29T10:47:59.570 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.b6zP7RLwDp 2024-03-29T10:47:59.585 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/data/mon.c.tgz 2024-03-29T10:47:59.585 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-29T10:47:59.640 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.KJCaQ7AvjX 2024-03-29T10:47:59.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.KJCaQ7AvjX 2024-03-29T10:47:59.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.KJCaQ7AvjX 2024-03-29T10:47:59.826 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.KJCaQ7AvjX is 212KB 2024-03-29T10:47:59.922 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.KJCaQ7AvjX 2024-03-29T10:47:59.937 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/data/mon.b.tgz 2024-03-29T10:47:59.937 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-29T10:47:59.955 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.8ZJN1qKa6a 2024-03-29T10:47:59.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8ZJN1qKa6a 2024-03-29T10:48:00.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.8ZJN1qKa6a 2024-03-29T10:48:00.161 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.8ZJN1qKa6a is 212KB 2024-03-29T10:48:00.255 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.8ZJN1qKa6a 2024-03-29T10:48:00.270 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:48:00.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:48:00.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:48:00.353 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:48:00.393 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:48:00.393 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:48:00.395 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi151/crash 2024-03-29T10:48:00.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:48:00.423 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi169/crash 2024-03-29T10:48:00.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:48:00.459 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:48:00.460 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:48:00.467 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:48:00.545 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:48:00.545 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-03-29T10:48:00.545 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34212.log 2024-03-29T10:48:00.545 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-03-29T10:48:00.545 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34212.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34492.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.34212.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34412.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34412.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34452.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.7.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34372.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:48:00.546 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34332.log 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34372.log.gz 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34292.log 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34332.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34292.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34332.log.gz 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34292.log.gz 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:48:00.547 INFO:teuthology.orchestra.run.smithi169.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:48:00.552 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40104.log 2024-03-29T10:48:00.552 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:48:00.552 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40026.log 2024-03-29T10:48:00.552 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40104.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40026.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40104.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38601.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39593.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40026.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39267.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38601.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38640.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39462.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39267.log.gz 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39710.log 2024-03-29T10:48:00.553 INFO:teuthology.orchestra.run.smithi151.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38640.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:gzip/var/log/ceph/ceph-client.admin.39710.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35109.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39710.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35109.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41354.log 2024-03-29T10:48:00.554 INFO:teuthology.orchestra.run.smithi151.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41354.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41549.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28896.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.tmp-client.admin.28896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41862.log 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28896.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-03-29T10:48:00.555 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38523.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41159.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38759.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41862.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38523.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38445.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41159.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38994.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38759.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38445.log.gz 2024-03-29T10:48:00.556 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40886.log: /var/log/ceph/ceph-client.admin.39792.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40182.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40182.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40182.log.gz 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41588.log 2024-03-29T10:48:00.557 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41432.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41588.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41667.log 86.3% 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-29T10:48:00.558 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40260.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41667.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39632.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40260.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39632.log.gz 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41120.log 2024-03-29T10:48:00.559 INFO:teuthology.orchestra.run.smithi151.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41706.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41120.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.3.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41120.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gz 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/ceph-client.admin.40925.log 2024-03-29T10:48:00.560 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40729.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38681.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40925.log.gz 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip 86.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz -- 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/ceph-client.admin.39189.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38681.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.40378.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38681.log.gz/var/log/ceph/ceph-client.admin.39189.log: 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39150.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39189.log.gz 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40964.log 2024-03-29T10:48:00.561 INFO:teuthology.orchestra.run.smithi151.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40964.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35006.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40964.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.35006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41081.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38562.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35006.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39671.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34715.log 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38562.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41081.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39671.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38562.log.gz 2024-03-29T10:48:00.562 INFO:teuthology.orchestra.run.smithi151.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39671.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.34715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41745.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34715.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38876.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0%/var/log/ceph/ceph-client.admin.41745.log: -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41745.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38876.log: /var/log/ceph/ceph-client.admin.40534.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38876.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38720.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose/var/log/ceph/ceph.log: -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2024-03-29T10:48:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.2.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38720.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/ceph-client.admin.38484.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38484.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39948.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38484.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr: 85.8%/var/log/ceph/ceph-client.admin.41393.log: -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38798.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.38406.log: /var/log/ceph/ceph-client.admin.38798.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-03-29T10:48:00.564 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38798.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41276.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40221.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41198.log: /var/log/ceph/ceph-client.admin.40221.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40221.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38954.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.39033.log: /var/log/ceph/ceph-client.admin.38954.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38954.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41237.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40612.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-03-29T10:48:00.565 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-03-29T10:48:00.566 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.41237.log: /var/log/ceph/ceph-client.admin.40690.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41237.log.gz 2024-03-29T10:48:00.566 INFO:teuthology.orchestra.run.smithi151.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-03-29T10:48:00.566 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:48:00.566 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.40456.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-03-29T10:48:00.584 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.c.log: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:48:12.062 INFO:teuthology.orchestra.run.smithi151.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:48:12.445 INFO:teuthology.orchestra.run.smithi151.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:48:12.590 INFO:teuthology.orchestra.run.smithi169.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:48:12.602 INFO:teuthology.orchestra.run.smithi169.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:48:16.032 INFO:teuthology.orchestra.run.smithi169.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:48:16.293 INFO:teuthology.orchestra.run.smithi151.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:48:16.430 INFO:teuthology.orchestra.run.smithi151.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:48:16.762 INFO:teuthology.orchestra.run.smithi169.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:48:50.280 INFO:teuthology.orchestra.run.smithi151.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:48:54.933 INFO:teuthology.orchestra.run.smithi151.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:48:54.935 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-29T10:48:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m54.456s 2024-03-29T10:48:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:user 2m33.720s 2024-03-29T10:48:54.935 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m5.537s 2024-03-29T10:49:06.730 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:49:06.731 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-29T10:49:06.731 INFO:teuthology.orchestra.run.smithi169.stderr:real 1m6.215s 2024-03-29T10:49:06.731 INFO:teuthology.orchestra.run.smithi169.stderr:user 1m57.491s 2024-03-29T10:49:06.731 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m4.381s 2024-03-29T10:49:06.732 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:49:06.732 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi151/log 2024-03-29T10:49:06.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:49:30.198 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi169/log 2024-03-29T10:49:30.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:49:49.306 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:49:49.508 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e36c1d657f9847278dcdb086029d81da Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:49:49.512 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:49:49.520 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:49:49.527 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:49:49.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:49:49.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:49:49.614 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:49:49.624 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:49:49.624 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:49:49.627 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:49:49.641 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:49:49.643 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:49:49.645 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:49:49.646 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-29T10:49:49.646 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +417us[ +417us] +/- 68ms 2024-03-29T10:49:49.646 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -182us[ -179us] +/- 68ms 2024-03-29T10:49:49.646 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -234us[ -230us] +/- 79ms 2024-03-29T10:49:49.646 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +420us[ +420us] +/- 68ms 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -198us[ -199us] +/- 68ms 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -239us[ -239us] +/- 79ms 2024-03-29T10:49:49.648 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:49:49.649 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:49:49.659 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:49:49.659 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:49:49.672 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:49:49.758 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:49:49.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:49:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:49.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:49:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:49.892 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:49:49.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:49:49.911 INFO:teuthology.task.internal:Duration was 1352.451225 seconds 2024-03-29T10:49:49.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:49:49.919 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:49:49.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:49:49.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:49:49.964 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:49:49.978 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:49:50.341 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:49:50.341 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-29T10:49:50.341 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:49:50.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-29T10:49:50.361 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:49:50.386 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:49:50.387 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:49:50.404 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:49:50.455 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:49:50.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:49:50.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:49:50.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:49:50.603 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:49:50.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:49:50.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:49:50.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:49:50.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:49:50.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:49:50.697 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-29T10:49:50.725 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-29T10:49:50.754 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:49:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:50.777 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:49:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:50.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:49:50.818 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:49:50.819 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi151 2024-03-29T10:49:50.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:49:50.856 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638/remote/smithi169 2024-03-29T10:49:50.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:49:50.892 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:49:50.893 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:49:50.895 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:49:50.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:49:50.954 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:49:50.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:49:50.995 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:49:50.996 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:49:50.998 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:49:51.010 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:49 /home/ubuntu/cephtest 2024-03-29T10:49:51.012 INFO:teuthology.orchestra.run.smithi169.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:49 /home/ubuntu/cephtest 2024-03-29T10:49:51.013 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:49:51.024 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:49:51.045 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:49:51.072 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-29T10:49:51.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:49:51.094 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-29T10:49:51.095 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:49:51.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:18.518909', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBx3xnHFbtJ25zGMuZ04L5GpSugqYgVK3jBBkv1e9NwE33Pz/SKbJjoF2qr6FsGun4tBX1imiEFrm8gjCWJMj7Q='} 2024-03-29T10:49:51.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:18.516968', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBY17LjcqhnrYdAm1fZk6fnGHYKtVuScVWQbsPTF73sszU4IAltJPgULm3CjqaD+OCMB9+GTv1bb7/2nO75Q508='} 2024-03-29T10:49:51.144 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-29T10:49:51.144 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:49:51.167 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-29T10:49:51.167 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:51.182 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-29T10:49:51.183 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:49:51.192 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-29T10:49:51.206 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-29T10:49:51.206 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:51.229 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-29T10:49:55.194 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:55.230 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:55.305 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-29T10:49:55.343 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-29T10:49:59.306 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:59.343 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:49:59.418 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-29T10:49:59.454 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-29T10:49:59.519 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-29T10:50:03.458 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:50:03.569 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-29T10:50:03.670 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-29T10:50:03.775 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1352.4512252807617 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e36c1d657f9847278dcdb086029d81da status: fail success: false 2024-03-29T10:50:03.775 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:50:03.860 INFO:teuthology.run:FAIL